Internship Project · OptiRisk Learning Systems · Jul 2022

Analysisstant
Automated NSE Stock-Market Data Collection, Storage & Analytics

Python Flask PostgreSQL BeautifulSoup Bootstrap 4 DataTables / jQuery CronJob
View on GitHub

About the project

Analysisstant is a Flask web application for viewing NSE (National Stock Exchange) equity-market data. A Python automation script pulls the NSE end-of-day closing inventory (bhavcopy) for the previous trading day, loads it into PostgreSQL, and a web layer lets registered users view market summaries, company details, and trade data. The entire collection pipeline is automated so the displayed information stays current without any manual effort.

What I built / how it worked

  • Automated daily ingestion — a Python script generates the NSE historical-equities URL from the previous day's date, downloads the bhavcopy .zip, and extracts the enclosed CSV — designed to run daily via CronJob after market close.
  • Smart holiday handling — a day is treated as non-working when the generated URL yields no file, with a logged, clean exit (no holiday calendar needed).
  • ETL-style storage — CSV normalized and bulk-loaded into PostgreSQL (psycopg2.copy_from) into a raw staging table; an ISIN-keyed Company master and processed closing-data table populated with SQL joins and upserts.
  • Enrichment via web scraping — scrapes each company's display name from the CDSL ISIN lookup page with BeautifulSoup and updates the Company table.
  • Web layer — Flask + Jinja2 + Bootstrap 4 with registration/login (Werkzeug hashed passwords) and role-gated pages for companies, NSE data, and profiles.
  • SQL-driven dashboard — market totals (entry count, aggregate trade quantity/value/trades) and top companies by trading activity for the latest trading day, computed in SQL.
  • Searchable tables — DataTables search/sort/pagination over company and NSE-data pages.

Feature highlights

  • Fully automated daily NSE bhavcopy acquisition, schedulable via CronJob.
  • Cleaned, structured PostgreSQL storage — raw staging, processed closing data, and an ISIN-keyed Company master.
  • CDSL web-scraped company-name enrichment with hashed-password auth.
  • SQL-driven analytics dashboard with market totals and top-company rankings.
← Back to Experience ← Back to all projects