65 skills found · Page 1 of 3
seekbytes / IPAGUI analyzer for deep-diving into PDF files. Detect malicious payloads, understand object relationships, and extract key information for threat analysis.
juanceresa / Sift KgTurn any collection of documents into a knowledge graph. Extract entities and relationships via LLM, deduplicate with your approval. Map domains, find hidden connections, spot patterns across documents — knowledge that persists and compounds, for you and your AI agents. All from the CLI.
jackboyla / GLiRELGeneralist and Lightweight Model for Relation Extraction (Extract any relationship types from text)
davidsbatista / BREDS"Bootstrapping Relationship Extractors with Distributional Semantics" (Batista et al., 2015) in EMNLP'15 - Python implementation
Orbifold / KnwlerKnwler is a lightweight, single-file Python tool that extracts structured knowledge graphs from documents using AI. Feed it a PDF or text file and receive a richly connected network of entities, relationships, and topics — complete with an interactive HTML report and exports ready for your favorite graph analytics platform.
opencog / RelexEnglish Dependency Relationship Extractor
OPTIMUM-LINKUP / Latest Optimum School SystemThis is the latest school management system. Available for all type of schools. will work on Phone, Laptop, Tabs, monitors - any screen size. It is available with 100% source code. The features are listed below: INSTALLATION Upload the downloaded zip file to your server in the public_html directory. Extract the zip file. Create a new database from your server mysql. Create user to the database and link the database to the user. Open the file database.php from the directory yourfolder/application/config/database.php. Fill up this information with your database hostname, database username, database password, database name respectively which you have created in the previous step. Now from server phpmyadmin go to your database. Select import and choose the file install.sql located in yourfolder/database/blank_db.sql (demo_db.sql for demo database) And you are ready to go now to browse the application Default admin credentials Email: admin@admin.com Password: admin ADMIN PANEL Managing User accounts (teacher, student, parent) Managing classes, subjects Managing exam, grades Managing exam marks Managing Loan Information Managing Computer Based Test (CBT) Sending exam marks via sms Managing students attendance Managing accounting, income and expenses Managing school events Managing Teachers Managing Libratrian Managing Accountant Manage Circular Manage Task Managing Parents Managing Alumni Managing Academic Sysllabus Managing Helpful Links Managing Help Desk Managing Front-End Information Managing School Session Attendance Reports Managing Staff ID Cards Records management. Notification board management. Management relationships between different type of users. Online Payment acceptance of FEE. Section Management. Reports generator. SMS Alerts. Managing Hostel Manager Managing library, dormitory, transport Messaging between other users Managing system settings (general, sms, language) Managing Media Subject management. Class management. Student payments management. Student behaviour management. Payments Overview. Subjects and assignments management. Fees management. Student assignment results management. Student search. Overdue students list. Student management. Student-Teacher interaction. And many more … TEACHERS Manage Students homework. Assign homework. Share homework on social networking sites (facebook). Manage classes. Manage Student Report. Generate Remarks on Student Reports. Generate Student Attendence. Subject management. Loan Application Class management. Student behaviour management. Subjects and assignments management. Student assignment results management. Student search. Student management. Student-Teacher interaction. Managing Helpful Links Managing Media Assignments Attendance Provide Daily Quotes Holidays Studennts Study Materials Message Noticeboard Transportations And many more… STUDENT PANEL Get class Routine Attempt Online Exam View Online Exam Result Get Exam Marks Message View Noticeboard Transportatio Receive SMS Get attendance status Get study materials / files from teacher Get payment invoice, Pay Online Communicate with teacher Managing Media accounts View Event Schedule, Notice and Holidays Get Helpful Links View Daily Quotes Contact Help Desks And many more …. PARENT PANEL View Children Marks View Children Class Routine Make payment View Payment Invoice Message Admin Message Teachers View Received Messkages Checkin kids progress. Parent-Teacher interaction. Get alerts from School Administration or Teachers. View events Noticeborad Todays Thought News Helpful Links Help Desk Receive SMS And many more … LIBRARIAN Add books Update books Record Lost Books Generate Reports on Books Subject Management. Loan Application Student Search. Student Management. Student-Librarian Interaction. View Helpful Links View Media Holidays Studennts Study Materials Message Transportations Noticeboard View Notification And many more …. ACCOUNTANTS Create Student Payments Students Payment Expenses Expenses Category Vew all Accountants Loan Application Todays Thought News Holidays Message k Noticeboard And many more …. HOSTEL MANAGER ViewAll Hostel Managers Manage Hostels Loan Application Todays Thought News Holidays Message Noticeboard And many more …. ………………………………………………………………………………………………… ADMIN PANEL DASHBORD ………………………………………………………………………………………………… Total number of students, teachers, librarian, accountants, hostel manager, alumni, parents and attendance of students for that day at a glance, Dashboard also holds a calendar for showing events, charts for various percentages of teachers, parents, students attendance, grades, students performances, etc. MANAGING SESSION From navigation go to manage session Add / edit / delete MANAGING ACADEMIC SYLLABUS From navigation go to manage academic syllabus Add / edit / delete MANAGING MEDIA From navigation go to manage media Add / edit / delete MANAGING STUDENTS Admit Students From navigation, go to students > admit students Fill up the necessary information Save student Admit Bulk Students From navigation, go to student > admit bulk student Download the blank Excel file Fill up the information Select class Upload the filled up Excel file Save Student Information From navigation go to student > student information Here you can see the students class wise If a class has sections then you can also browse the students as per class sections Student mark sheets From navigation go to student > student mark sheet Here you can see all the students marks class wise If the class has sections then you can also see them along with class MANAGING TEACHERS From navigation go to teacher Here you can see the list of teachers of your school in a tabular form To add a new teacher, click the top right button named add new teacher and fill up the information and save For editing or deleting a teacher information click the action button assigned to each entry of the table. That will bring two options for editing and deleting. Click on the required action editing and deleting MANAGING ACCOUNTANTS From navigation go to accountant Here you can see the list of accountants of your school in a tabular form To add a new accountant, click the top right button named add new accountant and fill up the information and save For editing or deleting a teacher information click the action button assigned to each entry of the table. That will bring two options for editing and deleting. Click on the required action editing and deleting MANAGING LIBRARIANS From navigation go to librarian Here you can see the list of librarians of your school in a tabular form To add a new librarian, click the top right button named add new librarian and fill up the information and save For editing or deleting a teacher information click the action button assigned to each entry of the table. That will bring two options for editing and deleting. Click on the required action editing and deleting MANAGING HOSTEL MANAGERS From navigation go to hostel manager Here you can see the list of hostel managers of your school in a tabular form To add a new hostel manager, click the top right button named add new hostel manager and fill up the information and save For editing or deleting a teacher information click the action button assigned to each entry of the table. That will bring two options for editing and deleting. Click on the required action editing and deleting MANAGING ALUMNI From navigation go to alumni Here you can see the list of alumni of your school in a tabular form To add a new alumni, click the top right button named add new alumni and fill up the information and save For editing or deleting a teacher information click the action button assigned to each entry of the table. That will bring two options for editing and deleting. Click on the required action editing and deleting MANAGING PARENTS From navigation go to parents Here you can see the list of parents of the students of your school in a tabular form To add a new parent, click the top right button named add new parent and fill up the information and save For editing or deleting a parent information click the action button assigned to each entry of the table. That will bring two options for editing and deleting. Click on the required action for editing and deleting MANAGING CLASSES From navigation go to > manage sections Add new class section for a class and assign teacher for each of them View the class sections in a tabular form class wise Edit and delete class section information MANAGING CLASS SECTION From navigation go to class > manage sections Add new class section for a class and assign a teacher for each of them View the class sections in a tabular for class wise Edit and deklete class section information MANAGING SUBJECTS From navigation go to subject If you have already added classes then under this you will see a list of the classes added. If you have not created classes, please create class first Here you can see the subjects class wise Add or edit or delete subjects MANAGING CLASS ROUTINE From navigation go to class routine View all the class routines in accordion Add class routine Click on the subject name on routine to edit and delete MANAGING DIALY STUDENT’S ATTENDENCE From navigation go to daily attendance Select the date and class and click manage attendance That will bring up the students name and attendance information in a tabular form To update the attendance status or for taking the attendance for that particular date of that particular class which you have selected earlier, click the button named update attendance Put the status for all at once and click save changes MANAGING EXAMS Exam list From navigation, go to exam > exam list Add an exam for all Edit and delete exam Exam grade From navigation go to exam > exam grades Add exam grades as per the requirements of your institution Edit or delete exam grades Manage exam marks From navigation go to > manage marks Select exam, class and subject and click manage marks for changing or updating marks That will bring up the form for updating the students marks for that particular subject Enter the marks and click update Sending exam marks by SMS From navigation go to exam > send mark by SMS Select exakm and class and receive (students/parent) Click the button named send mark via SMS That will send SMS with the marks for that exam you have selected if a SMS service is already activated MANAGING PAYMENTS From navigation go to payment Add invoice and take manual payment multiple time under the same invoice If a payment is due, then an option will be there for taking the payment in the action button of the table that contains the list of all the invoices with the basic information. Edit or delete invoice LOAN MANAGEMENT From navigation go to loan application See all the applied loans Click on apply loan Fill forms to apply Wait for loan approval COMPUTER BASED TEST (CBT) From navigation, go to Manage CBT Click on Add Exam Set Class, Exam Time, Exam Duration, Subject, Question Count and Session Click on continue to Add Questions Click on List Exams to View Exams Click on View Result to View Exams Scores ACCOUNTING Incomes From navigation, go to accounting > incomes Here you can see all the incomes for your school that means students fee in a tabular form with their payment time and amount EXPENSES From navigation, go to accounting > expenses Add expenses for the school Edit or delete them GENERATING STAFF IDCARD Teacher, librarian, accounant, hostel manager From navigation, go to staff > ID CARD Here you can you will see a button asking you to click generate ID CARD EXPENSE CATEGORY From navigation, go to accounting > expense category Add expense category Edit or delete them MANAGING BOOKS From navigation go to library Add books Edit or delete them MANAGING TRANSPORT From navigation go to transport Add transport information Edit or delete them MANAGING DORMITORY From navigation go to dormitory Add / edit / delete MANAGING ASSIGNMENT From navigation go to assignment Add / edit / delete MANAGING HOLIDAYS From navigation go to holiday Add / edit / delete MANAGING TODAY’S THOUGHT From navigation go to today’s thought Add / edit / delete MANAGING CIRCULAR From navigation go to circular Add / edit / delete MANAGING SCHOOL CLUBS From navigation go to school club Add / edit / delete MANAGING TASK From navigation go to task manager Add / edit / delete MANAGING HELPFUL LINK From navigation go to Helpful Links Add / edit / delete MANAGING ENQUIRY From navigation go to enquiry Add / edit / delete MANAGING ENQUIRY CATEFORY From navigation go to enquiry category Add / edit / delete MANAGING HELP DESK From navigation go to task Helpdesk Add / edit / delete NOTICEBOARD From navigation go to notice board Add / edit / delete them For sending the notice to all as SMS, yes while creating the notice This will send SMS to all the users about that notice PRIVATE MESSAGING From navigation, go to message Admin can send message to all users For sending message, select user and type the message and click send You can also see all the message sent to you or sent from you SYSTEM SETTINGS From navigation go to settings > general settings You can change basic system settings here and also can select language You can also upload logo from here THEME SETTINGS From navigation go to setting > general settings On the right of the page there is a panel named theme settings You find several skin options for you application Select you desire one to make changes SMS SETTINGS From navigation go to settings > sms settings Here you will find 2 SMS services, one is Clickatell and another is Twilio You have to activate a service first Then put the necessary information for a service Visit https://www.twilio.com/user/acount/settings/international /sms LANGUAGE SETTINGS From navigation go to setting > language settings Change phrase or add new phrase for a particular language Add new language MANAGE BANNER SETTINGS From navigation go to setting > banner settings Add / edit / delete MANAGE FRONT END SETTINGS From navigation go to setting > front end settings Add / edit / delete MANAGE NEWS SETTINGS From navigation go to setting > news settings Add / edit / delete ACCOUNT SETTINGS From navigation go to account Change basic account information Update your password Change profile image ……………………………………………………………………………………………….. TEACHER PANEL DASHBOARD ………………………………………………………………………………………………. Total number of students, parents and attendance of students for that day at a glance Dashboard also holds a calendar for showing events. MANAGING STUDENTS Admit students From navigation go to student > admit student Fill up the necessary information Save student Student information From navigation go to student > student information Here you can see the student class wise If a class has sections then you can also browse the students as per class sections Student mark sheets From navigation go to student > student mark sheet Here you can see all the students marks class wise If the class has sections then you can also see them along with class MANAGING DAILY STUDENT’S ATTENDANCE From navigation go to daily attendance Select the date and class and click mange attendance That will bring up the students name and attendance information in a tabular form To update the attendance status or for taking the attendance for that particular date of that particular class which you have selected earlier, click the button named update attendance Put the status for all at once and click save changes MANAGING DAILY STUDENT’S ATTENDANCE From navigation go to daily attendance Select the date and class and click manage attendance That will bring up the students name and attendance information in a tabular form To update the attendance status or for taking the attendance for that particular date of that particular class which you have selected earlier, click the button named update attendance Put the status for all at once and click save changes MANAGING ASSIGNMENT From navigation go to assignment That will bring up the assignemnt page in a tabular form, you can click on add assignment on left corner of the page to add assignment. MANAGING CLASSES From navigation go to > manage sections Add new class section for a class and assign teacher for each of them View the class sections in a tabular form class wise Edit and delete class section information MANAGING CLASS SECTION From navigation go to class > manage sections Add new class section for a class and assign a teacher for each of them View the class sections in a tabular for class wise Edit and deklete class section information MANAGING SUBJECTS From navigation go to subject If you have already added classes then under this you will see a list of the classes added. If you have not created classes, please create class first Here you can see the subjects class wise Add or edit or delete subjects MANAGING CLASS ROUTINE From navigation go to class routine View all the class routines in accordion Add class routine Click on the subject name on routine to edit and delete MANAGING DIALY STUDENT’S ATTENDENCE From navigation go to daily attendance Select the date and class and click manage attendance That will bring up the students name and attendance information in a tabular form To update the attendance status or for taking the attendance for that particular date of that particular class which you have selected earlier, click the button named update attendance Put the status for all at once and click save changes MANAGING EXAMS Manage exam marks From navigation go to > manage marks Select exam, class and subject and click manage marks for changing or updating marks That will bring up the form for updating the students marks for that particular subject Enter the marks and click update MANAGING HELPFUL LINK From navigation go to Helpful Links Add / edit / delete NEWS From navigation go to view news View all the uploaded news TODAY’S THOUGHT From navigation go to today’s thought View all the uploaded today’s thought HOLIDAY DATES From navigation go to holiday View all the holiday with their respectives dates ……………………………………………………………………………………………………. STUDENT PANEL DASHBOARD ……………………………………………………………………………………………………. Total number of students, teachers, parents and attendance of students for that day at a glance, dashboard also holds a calendar for showing event CLASS ROUTINE Form navigation go to class routine View the class routine of the logged in student EXAM MARKS From navigation go to exam > manage marks Select exam and subject See the mark for the selected exam in the selected subject COMPUTER BASED TEST (CBT) From navigation go to online CBT See all the uploaded test for your class Attemtp the uploaded test View your results STUDY MATERIALS From navigation go to study materials See all the uploaded study materials for your class Download the materials ASSIGNMENT From navigation go to assignment See all the uploaded assignments for your class Download the assignment MEDIA From navigation go to media See all the uploaded media for your class Download or watch media NEWS From navigation go to view news View all the uploaded news TODAY’S THOUGHT From navigation go to today’s thought View all the uploaded today’s thought HOLIDAY DATES From navigation go to holiday View all the holiday with their respectives dates HELPFUL LINKS From navigation go to helpful links View all the helpful links HELP DESK From navigation go to help desk Submit or create help desk to the administrator STUDY MATERIALS From navigation go to study material See all the uploaded study material for your class Download the study material PAYMENT / PAY WITH PAYPAL From navigation go to payment See the list of invoices Pay online with paypal for the unpaid invoices COMMUNICATE WITH TEACHERS / ADMIN From navigation go to message Send new message to teachers and admin Get the sent message to you ………………………………………………………………………………………………...... ACCOUNTANT PANEL DASHBOARD ………………………………………………………………………………………………….. Total number of students, accountants, parents and attendance of student for that day at a glance. Dashboard also holds a calendar for showing events. MANAGING PAYMENTS From navigation go to payment Add invoice and take manual payment multiple time under the same invoice If a payment is due, then an option will be there for taking the payment in the action button of the table that contains the list of all the invoices with the basic information. Edit or delete invoice LOAN MANAGEMENT From navigation go to loan application See all the applied loans Click on apply loan Fill forms to apply Wait for loan approval MESSAGING From navigating go to message Send message to teachers and admin Get the message sent to you NEWS From navigation go to view news View all the uploaded news TODAY’S THOUGHT From navigation go to today’s thought View all the uploaded today’s thought HOLIDAY DATES From navigation go to holiday View all the holiday with their respectives dates HELPFUL LINKS From navigation go to helpful links View all the helpful links HELP DESK From navigation go to help desk Submit or create help desk to the administrator TRANSPORTATION From navigation go to transportation View transportation available ………………………………………………………………………………………………...... LIBRARIAN PANEL DASHBOARD ………………………………………………………………………………………………….. Total number of students, librarian, parents and attendance of student for that day at a glance. Dashboard also holds a calendar for showing events. MANAGING BOOKS From navigation go to library Add books Edit or delete them LOAN MANAGEMENT From navigation go to loan application See all the applied loans Click on apply loan Fill forms to apply Wait for loan approval MESSAGING From navigating go to message Send message to teachers and admin Get the message sent to you NEWS From navigation go to view news View all the uploaded news TODAY’S THOUGHT From navigation go to today’s thought View all the uploaded today’s thought HOLIDAY DATES From navigation go to holiday View all the holiday with their respectives dates HELPFUL LINKS From navigation go to helpful links View all the helpful links HELP DESK From navigation go to help desk Submit or create help desk to the administrator TRANSPORTATION From navigation go to transportation View transportation available ………………………………………………………………………………………………...... HOSTEL MANAGER PANEL DASHBOARD ………………………………………………………………………………………………….. Total number of students, hostel managers, parents and attendance of student for that day at a glance. Dashboard also holds a calendar for showing events. MANAGING DORMITORY From navigation go to dormitory Add / edit / delete LOAN MANAGEMENT From navigation go to loan application See all the applied loans Click on apply loan Fill forms to apply Wait for loan approval MESSAGING From navigating go to message Send message to teachers and admin Get the message sent to you NEWS From navigation go to view news View all the uploaded news TODAY’S THOUGHT From navigation go to today’s thought View all the uploaded today’s thought HOLIDAY DATES From navigation go to holiday View all the holiday with their respectives dates HELPFUL LINKS From navigation go to helpful links View all the helpful links HELP DESK From navigation go to help desk Submit or create help desk to the administrator TRANSPORTATION From navigation go to transportation View transportation available ………………………………………………………………………………………………...... PARENT PANEL DASHBOARD ………………………………………………………………………………………………….. Total number of students, teachers, parents and attendance of student for that day at a glance. Dashboard also holds a calendar for showing events. CHILDREN MARKS From navigation go to exam marks See the mark of your children individually One parent can have multiple children PAYMENTS From navigation go to exam > payment View the invoices of your children and individually Make payment via paypal online CLASS ROUTINE From navigation go to class routine Get the class routine for each of your child separately MESSAGING From navigating go to message Send message to teachers and admin Get the message sent to you NEWS From navigation go to view news View all the uploaded news TODAY’S THOUGHT From navigation go to today’s thought View all the uploaded today’s thought HOLIDAY DATES From navigation go to holiday View all the holiday with their respectives dates HELPFUL LINKS From navigation go to helpful links View all the helpful links HELP DESK From navigation go to help desk Submit or create help desk to the administrator
muchdogesec / Txt2stixExtracts IoCs, TTPs and the relationships between them. Outputs a STIX 2.1 bundle.
greenelab / SnorkelingExtracting biomedical relationships from literature with Snorkel 🏊
michalgm / CorpwatchapiThe CorpWatch API uses automated parsers to extract the subsidiary relationship information from Exhibit 21 of companies' 10-K filings with the SEC and provides a free, well-structured interface for programs to query and process the data.
ARUN-S-CODER / Table Recognition ProjectExtract tables from invoice images, process text using OCR, extract entities and relationships using LLM and traditional methods, and construct a visual knowledge graph.
fractalego / PynsettA programmable relation extraction tool
HlG4399 / FRSVTI implemented the fllowing article by Matlab.Refrence:Oh T H, Matsushita Y, Tai Y W, et al. Fast Randomized Singular Value Thresholding for Low-rank Optimization[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2015, PP(99):1-1.Abstract:Rank minimization can be converted into tractable surrogate problems, such as Nuclear Norm Minimization (NNM) and Weighted NNM (WNNM). The problems related to NNM, or WNNM, can be solved iteratively by applying a closed-form proximal operator, called Singular Value Thresholding (SVT), or Weighted SVT, but they suffer from high computational cost of Singular Value Decomposition (SVD) at each iteration. We propose a fast and accurate approximation method for SVT, that we call fast randomized SVT (FRSVT), with which we avoid direct computation of SVD. The key idea is to extract an approximate basis for the range of the matrix from its compressed matrix. Given the basis, we compute partial singular values of the original matrix from the small factored matrix. In addition, by developping a range propagation method, our method further speeds up the extraction of approximate basis at each iteration. Our theoretical analysis shows the relationship between the approximation bound of SVD and its effect to NNM
dalunacrobate / BioIGBioIG is a module that allows you to extract personnals informations from an instagram profile bio such as : Religion, Hobbies, Ethnicity, Emails, Paypal.me/, Snapchat, Twitter, Best Friends, Age, Location, Love Relationship, Love Relationship date, Facebooks, Astrologic Sign, Sexuality.
zoebear / RadiaRadia is a tool designed to create an interactive and immerse environment to visualize code, and to augment the task of reverse engineering binaries. The tool takes decompiled binaries extracted through IDA Pro, and visualizes the call graph in 3D space as a force directed graph. Radia tags functions that could be potential problems, as well as giving the user specific insight into individual nodes and their contents and relationships to other functions. In the end, the hope is to improve the available tools for auditing code to readily identify and remediate problems, and ultimately, to make code less vulnerable to exploitation by malware.
stucco / Relation ExtractorExtract relationships between cyber security entities within unstructured text
google-research-datasets / WebREDWebRED is a large and diverse manually annotated dataset for extracting relationships from a variety of text found on the World Wide Web.
schoobani / Novel GraphExtracts character interaction graphs from novels using LLMs, mapping relationships, descriptions, and interactions.
word-fish / Wordfish Pythonextract relationships from standardized terms from corpus of interest with deep learning :fish:
ultranet1 / APACHE AIRFLOW DATA PIPELINESProject Description: A music streaming company wants to introduce more automation and monitoring to their data warehouse ETL pipelines and they have come to the conclusion that the best tool to achieve this is Apache Airflow. As their Data Engineer, I was tasked to create a reusable production-grade data pipeline that incorporates data quality checks and allows for easy backfills. Several analysts and Data Scientists rely on the output generated by this pipeline and it is expected that the pipeline runs daily on a schedule by pulling new data from the source and store the results to the destination. Data Description: The source data resides in S3 and needs to be processed in a data warehouse in Amazon Redshift. The source datasets consist of JSON logs that tell about user activity in the application and JSON metadata about the songs the users listen to. Data Pipeline design: At a high-level the pipeline does the following tasks. Extract data from multiple S3 locations. Load the data into Redshift cluster. Transform the data into a star schema. Perform data validation and data quality checks. Calculate the most played songs for the specified time interval. Load the result back into S3. dag Structure of the Airflow DAG Design Goals: Based on the requirements of our data consumers, our pipeline is required to adhere to the following guidelines: The DAG should not have any dependencies on past runs. On failure, the task is retried for 3 times. Retries happen every 5 minutes. Catchup is turned off. Do not email on retry. Pipeline Implementation: Apache Airflow is a Python framework for programmatically creating workflows in DAGs, e.g. ETL processes, generating reports, and retraining models on a daily basis. The Airflow UI automatically parses our DAG and creates a natural representation for the movement and transformation of data. A DAG simply is a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies. A DAG describes how you want to carry out your workflow, and Operators determine what actually gets done. By default, airflow comes with some simple built-in operators like PythonOperator, BashOperator, DummyOperator etc., however, airflow lets you extend the features of a BaseOperator and create custom operators. For this project, I developed several custom operators. operators The description of each of these operators follows: StageToRedshiftOperator: Stages data to a specific redshift cluster from a specified S3 location. Operator uses templated fields to handle partitioned S3 locations. LoadFactOperator: Loads data to the given fact table by running the provided sql statement. Supports delete-insert and append style loads. LoadDimensionOperator: Loads data to the given dimension table by running the provided sql statement. Supports delete-insert and append style loads. SubDagOperator: Two or more operators can be grouped into one task using the SubDagOperator. Here, I am grouping the tasks of checking if the given table has rows and then run a series of data quality sql commands. HasRowsOperator: Data quality check to ensure that the specified table has rows. DataQualityOperator: Performs data quality checks by running sql statements to validate the data. SongPopularityOperator: Calculates the top ten most popular songs for a given interval. The interval is dictated by the DAG schedule. UnloadToS3Operator: Stores the analysis result back to the given S3 location. Code for each of these operators is located in the plugins/operators directory. Pipeline Schedule and Data Partitioning: The events data residing on S3 is partitioned by year (2018) and month (11). Our task is to incrementally load the event json files, and run it through the entire pipeline to calculate song popularity and store the result back into S3. In this manner, we can obtain the top songs per day in an automated fashion using the pipeline. Please note, this is a trivial analyis, but you can imagine other complex queries that follow similar structure. S3 Input events data: s3://<bucket>/log_data/2018/11/ 2018-11-01-events.json 2018-11-02-events.json 2018-11-03-events.json .. 2018-11-28-events.json 2018-11-29-events.json 2018-11-30-events.json S3 Output song popularity data: s3://skuchkula-topsongs/ songpopularity_2018-11-01 songpopularity_2018-11-02 songpopularity_2018-11-03 ... songpopularity_2018-11-28 songpopularity_2018-11-29 songpopularity_2018-11-30 The DAG can be configured by giving it some default_args which specify the start_date, end_date and other design choices which I have mentioned above. default_args = { 'owner': 'shravan', 'start_date': datetime(2018, 11, 1), 'end_date': datetime(2018, 11, 30), 'depends_on_past': False, 'email_on_retry': False, 'retries': 3, 'retry_delay': timedelta(minutes=5), 'catchup_by_default': False, 'provide_context': True, } How to run this project? Step 1: Create AWS Redshift Cluster using either the console or through the notebook provided in create-redshift-cluster Run the notebook to create AWS Redshift Cluster. Make a note of: DWN_ENDPOINT :: dwhcluster.c4m4dhrmsdov.us-west-2.redshift.amazonaws.com DWH_ROLE_ARN :: arn:aws:iam::506140549518:role/dwhRole Step 2: Start Apache Airflow Run docker-compose up from the directory containing docker-compose.yml. Ensure that you have mapped the volume to point to the location where you have your DAGs. NOTE: You can find details of how to manage Apache Airflow on mac here: https://gist.github.com/shravan-kuchkula/a3f357ff34cf5e3b862f3132fb599cf3 start_airflow Step 3: Configure Apache Airflow Hooks On the left is the S3 connection. The Login and password are the IAM user's access key and secret key that you created. Basically, by using these credentials, we are able to read data from S3. On the right is the redshift connection. These values can be easily gathered from your Redshift cluster connections Step 4: Execute the create-tables-dag This dag will create the staging, fact and dimension tables. The reason we need to trigger this manually is because, we want to keep this out of main dag. Normally, creation of tables can be handled by just triggering a script. But for the sake of illustration, I created a DAG for this and had Airflow trigger the DAG. You can turn off the DAG once it is completed. After running this DAG, you should see all the tables created in the AWS Redshift. Step 5: Turn on the load_and_transform_data_in_redshift dag As the execution start date is 2018-11-1 with a schedule interval @daily and the execution end date is 2018-11-30, Airflow will automatically trigger and schedule the dag runs once per day for 30 times. Shown below are the 30 DAG runs ranging from start_date till end_date, that are trigged by airflow once per day. schedule
