KN
The graph itself is just rendered after de-serializing the Serialized_dag
Size: a a a
2020 August 04
KN
yea) I mean that process add to DB info about new DAG at the first time?
Oh yeah, the scheduler continually parses new DAG files and stores the new DAGs in both "dag" table and in serialized_dag table
YV
Oh yeah, the scheduler continually parses new DAG files and stores the new DAGs in both "dag" table and in serialized_dag table
got it, it done by scheduler, not by webserver
KN
Yeah -- this was done so that Webserver becomes light-weight
KN
And parsing of DAGs only happens at one place -- "Scheduler"
KN
Webserver just reads everything from the DB
YV
And parsing of DAGs only happens at one place -- "Scheduler"
cool, make sense
KN
it no longer needs access to DAG files
YV
very cool and informative talk ) thank you
KN
Thanks Iuliia :)
IS
Время вопросов :)
YV
@kaxil one more question from me) what about ML, do you think that Airflow will cover 'ML-fields' in feature or better to focus on things like ML Flow https://mlflow.org/ if need to care about ML tasks?
YV
I remember was some mails about it in dev airflow maillist, but not sure that was a result of discussion
KN
Thats a great question. One of the main motivation of the Functional DAGs was to support ML use-cases where the Data Scientist don't need to worry about Airflow syntaxes instead they can just write the same Python Code.
Twitter uses it interally for their ML Workflows. Here is a blog post on how they do it: https://blog.twitter.com/engineering/en_us/topics/insights/2018/ml-workflows.html
Twitter uses it interally for their ML Workflows. Here is a blog post on how they do it: https://blog.twitter.com/engineering/en_us/topics/insights/2018/ml-workflows.html
KN
And also we want to support more ML use-cases with Airflow so we are planning to have a native View to see how the models performed in the previous runs via some user-configurable metric like rmse value. However if they would also want to scheduler other pipelines ETL or anything else, Airflow would be ideal
KN
I would say though that MLFlow specializes in ML Worflows so if a company only wants to have ML pipeline, they would be better with MLFlow instead of Airflow
IS
Сейчас в эфире @switowski
С темой: Wait, IPython can do that?!
Во время доклада отмечайте спикера и задавайте вопросы в чате на английском, он вам ответит.
Вопросы на русском также можно писать, их модератор задаст в прямом эфире после доклада.
Если хотите задать вопрос лично в эфире, проверяйте свою камеру и гарнитуру и пишите мне в личку :)
С темой: Wait, IPython can do that?!
Во время доклада отмечайте спикера и задавайте вопросы в чате на английском, он вам ответит.
Вопросы на русском также можно писать, их модератор задаст в прямом эфире после доклада.
Если хотите задать вопрос лично в эфире, проверяйте свою камеру и гарнитуру и пишите мне в личку :)
YV
I would say though that MLFlow specializes in ML Worflows so if a company only wants to have ML pipeline, they would be better with MLFlow instead of Airflow
got it) thank you for the answers