Data engineering for pipelines, OCR and data platforms
Turning scattered sources into reliable data
Many projects fail not because of the model, but because of the data that feeds it: scattered, in different formats, some only as scanned PDFs. We build the pipelines that turn this into reliable, structured data.
That includes crawling and ingestion, OCR and document processing, extraction with LLMs, data quality and the platforms all of this runs on.
The result is data that downstream systems, analytics and AI features can build on.
The typical situation
At Blue Elephant Energy, public announcements about solar parks were scattered across the web, in different formats and some only as scanned PDFs. Working manually, the acquisition team could not keep up with the volume. Patent law firms want to make large document collections searchable.
How we approach it
- Crawling: parallel extraction processes that search millions of websites for relevant publications.
- OCR with AWS Textract, so scanned documents become machine-readable.
- Rule-based pre-filters that remove false positives before any LLM costs arise.
- Extraction with LLMs: A language model pulls out the relevant information and structures it for assessment.
- Vector search: For Patent Studio we segment PDFs and build a chunking and embedding pipeline for similarity search.
The Blue Elephant Energy pipeline has 3 stages: crawl, OCR, LLM. It was built in 2024 and has been running independently in production since handover. 7,235 patents have gone through the opt-out process via Patent Studio.
How it works
We approach data projects in three phases. Each phase ends with a result you use to decide whether and how to continue.
- Consulting and architecture, from one week: We work out which sources exist, which formats the data comes in and which systems will use the results later. The result is a decision paper with architecture, options and an effort estimate.
- Prototyping and pre-study, typically 2–6 weeks: We test on sample documents how well OCR, pre-filters and extraction work, and assess what the build will take. The result is a clickable prototype, a risk assessment and a solid implementation plan.
- Engineering and operations, months to years: We build the pipeline, take it into production and hand it over so that it keeps running without us. The result is software in production, documented and ready to hand over to your team.
The Blue Elephant Energy pipeline was built in three months in 2024 and has been running independently since handover. You can find an overview of the phases under Services.
Technologies
These are the technologies we use in data projects:
- Crawling and ingestion: Node.js and TypeScript, parallel extraction processes
- OCR: AWS Textract for scanned documents and PDFs
- Extraction with LLMs: OpenAI and LangChain
- Vector search: chunking and embedding pipeline with Azure AI Search
- Operations: Docker, AWS and Azure
- Tools for business teams: React, Next.js and Tailwind
For analysis across large datasets we combine classic data analysis, statistical methods and AI-assisted techniques, as in the project with T-Systems.
Frequently asked questions
Can you process scanned PDFs and inconsistent documents?
Yes. At Blue Elephant Energy, public announcements came in different formats, some only as scanned PDFs. An OCR stage with AWS Textract makes them machine-readable. Rule-based pre-filters remove obvious false positives. A language model then pulls out the relevant information and structures it for the business team to assess.
How do you keep language model costs under control?
With pre-filters before the LLM stage. In the Blue Elephant Energy pipeline, rule-based filters eliminate obvious false positives before any LLM costs arise. The language model only evaluates the remaining documents. The pipeline has three stages: crawl, OCR and LLM. Each stage has one clear job, so you can see where the effort goes.
Does the pipeline keep running without you after the project?
That is the goal. The Blue Elephant Energy pipeline has been running independently in production since the project ended. We are no longer actively involved; the system was handed over cleanly to the client. One experienced engineer built it in three months. A clearly scoped problem does not always need a large team.
How do you make large document collections searchable?
With vector search. For Patent Studio we split PDFs into sections after OCR and build a chunking and embedding pipeline. On top of it runs a similarity search across large document collections with Azure AI Search. 7,235 patents have gone through the opt-out process via Patent Studio. All AI features are built so that sensitive client data stays under control.
- Crawling and ingestion from any source
- OCR and document processing
- Extraction and enrichment with LLMs
- Data quality and validation
- Migrations between systems
- Vector search and knowledge bases
- Structured data from unstructured sources
- Pipelines that run independently in production
- A reliable foundation for AI features
- Consistency in large-volume migrations
- Access for downstream systems