Search Engine and Web Crawler in Python

• Implement a web crawler
• japanese morphological analysis using janome
• Implement search engine
• Store in MongoDB
• Web frontend using Flask

More details are avairable from My Tech Blog(Japanese).

Requirements

• Python 3.5

Setup

1. Clone repository

   $ git clone git@github.com:mejiro/SearchEngine.git
   

2. Install python packages

   $ cd SearchEngine
   $ pip install -r requirements.txt -c constraints.txt
   

3. MongoDB settings

4. Run

   $ python manage.py crawler # build a index
   $ python manage.py webpage # access to http://127.0.0.1:5000