← Back to blog

From PDF to Structured Data: Building an Extraction Pipeline with Laravel, PostgreSQL pgvector, and Embeddings for Internal RAG

2026-07-19

From PDF to Structured Data: Building an Extraction Pipeline with Laravel, PostgreSQL pgvector, and Embeddings for Internal RAG

Retrieval-Augmented Generation (RAG) has become the standard pattern for building AI applications that leverage proprietary data. But before your LLM can answer questions about internal documents, you need a robust pipeline that transforms unstructured PDFs into searchable vector embeddings. In this post, I'll walk you through building this pipeline using Laravel, PostgreSQL with the pgvector extension, and OpenAI embeddings.

Why This Stack Matters

Most RAG tutorials use Python and Pinecone. That's fine for prototypes, but enterprise teams often need something different: a stack their existing developers understand, infrastructure they already operate, and data that never leaves their controlled environment. Laravel with PostgreSQL pgvector checks all these boxes.

Laravel provides excellent queue processing, job scheduling, and API scaffolding. PostgreSQL with pgvector gives you ACID-compliant vector storage without introducing another database. Combined, you get production-grade RAG infrastructure that your team can maintain long-term.

Architecture Overview

Our pipeline has five stages:

Setting Up PostgreSQL with pgvector

First, ensure your PostgreSQL instance has pgvector installed. On most systems:

# Ubuntu/Debian
sudo apt install postgresql-16-pgvector

# Then in psql
CREATE EXTENSION vector;

For Laravel, install the pgvector PHP bindings and configure your database connection in config/database.php:

'pgsql' => [
    'driver' => 'pgsql',
    'host' => env('DB_HOST', '127.0.0.1'),
    'port' => env('DB_PORT', '5432'),
    'database' => env('DB_DATABASE', 'forge'),
    'username' => env('DB_USERNAME', 'forge'),
    'password' => env('DB_PASSWORD', ''),
    'charset' => 'utf8',
    'prefix' => '',
    'prefix_indexes' => true,
    'search_path' => 'public',
    'sslmode' => 'prefer',
],

Install the vector type support package:

composer require ankane/pgvector-php

Database Schema Design

Create migrations for documents and their chunked embeddings:

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function up(): void
    {
        Schema::create('documents', function (Blueprint $table) {
            $table->id();
            $table->string('title');
            $table->string('file_path');
            $table->string('source_url')->nullable();
            $table->integer('total_pages')->default(0);
            $table->jsonb('metadata')->nullable();
            $table->timestamps();
        });

        Schema::create('document_chunks', function (Blueprint $table) {
            $table->id();
            $table->foreignId('document_id')->constrained()->onDelete('cascade');
            $table->integer('chunk_index');
            $table->text('content');
            $table->integer('page_number')->nullable();
            $table->vector('embedding', 1536); // OpenAI text-embedding-3-small
            $table->timestamps();

            $table->index('document_id');
        });
    }

    public function down(): void
    {
        Schema::dropIfExists('document_chunks');
        Schema::dropIfExists('documents');
    }
};

Note the vector column type with dimension 1536. This matches OpenAI's text-embedding-3-small model. If you use text-embedding-3-large, change this to 3072.

PDF Text Extraction Service

For PDF parsing, I recommend smalot/pdfparser as a pure PHP solution, or shell out to pdftotext for better accuracy with complex layouts:

composer require smalot/pdfparser

Here's the extraction service:

<?php

namespace App\Services;

use Smalot\PdfParser\Parser;
use Illuminate\Support\Facades\Storage;

class PdfExtractionService
{
    private Parser $parser;

    public function __construct()
    {
        $this->parser = new Parser();
    }

    public function extract(string $filePath): array
    {
        $fullPath = Storage::path($filePath);
        $pdf = $this->parser->parseFile($fullPath);
        $details = $pdf->getDetails();
        
        $pages = $pdf->getPages();
        $pageContents = [];
        
        foreach ($pages as $index => $page) {
            $text = $page->getText();
            // Normalize whitespace and fix hyphenation
            $text = $this->normalizeText($text);
            
            $pageContents[] = [
                'page_number' => $index + 1,
                'content' => $text,
                'character_count' => strlen($text),
            ];
        }

        return [
            'title' => $details['Title'] ?? basename($filePath),
            'total_pages' => count($pages),
            'pages' => $pageContents,
        ];
    }

    private function normalizeText(string $text): string
    {
        $text = preg_replace('/-\n/', '', $text);
        $text = preg_replace('/\s+/', ' ', $text);
        return trim($text);
    }
}

Intelligent Chunking with Overlap

Simple fixed-size chunking destroys context at boundaries. Implement semantic chunking with sentence-aware splitting and overlap:

<?php

namespace App\Services;

class ChunkingService
{
    public function __construct(
        private int $chunkSize = 1000,
        private int $overlapSize = 200
    ) {}

    public function chunk(array $pages): array
    {
        $fullText = implode("\n\n", array_column($pages, 'content'));
        $sentences = $this->splitIntoSentences($fullText);
        
        $chunks = [];
        $currentChunk = '';
        $currentPage = 1;
        $chunkIndex = 0;
        $sentenceIndex = 0;

        while ($sentenceIndex < count($sentences)) {
            $sentence = $sentences[$sentenceIndex];
            
            if (strlen($currentChunk) + strlen($sentence) > $this->chunkSize && strlen($currentChunk) > 0) {
                $chunks[] = [
                    'content' => trim($currentChunk),
                    'chunk_index' => $chunkIndex,
                    'page_number' => $currentPage,
                ];
                
                // Overlap: keep last portion for context
                $currentChunk = $this->getOverlap($currentChunk);
                $chunkIndex++;
            }

            $currentChunk .= ' ' . $sentence;
            $sentenceIndex++;
        }

        // Don't lose the final chunk
        if (strlen(trim($currentChunk)) > 100) {
            $chunks[] = [
                'content' => trim($currentChunk),
                'chunk_index' => $chunkIndex,
                'page_number' => $currentPage,
            ];
        }

        return $chunks;
    }

    private function splitIntoSentences(string $text): array
    {
        return preg_split('/(?<=[.!?])\s+/', $text, -1, PREG_SPLIT_NO_EMPTY);
    }

    private function getOverlap(string $chunk): string
    {
        $words = explode(' ', $chunk);
        $overlapWords = array_slice($words, -intdiv($this->overlapSize, 5));
        return implode(' ', $overlapWords) . ' ';
    }
}

Embedding Generation Service

Call OpenAI's embedding API with proper batching and retry logic:

<?php

namespace App\Services;

use Illuminate\Support\Facades\Http;
use Illuminate\Support\Facades\Log;

class EmbeddingService
{
    private string $apiKey;
    private string $model;

    public function __construct()
    {
        $this->apiKey = config('services.openai.api_key');
        $this->model = config('services.openai.embedding_model', 'text-embedding-3-small');
    }

    public function generate(string $text): array
    {
        // OpenAI has a token limit per request, truncate if needed
        $text = $this->truncateToTokenLimit($text, 8191);

        $response = Http::withHeaders([
            'Authorization' => 'Bearer ' . $this->apiKey,
            'Content-Type' => 'application/json',
        ])->post('https://api.openai.com/v1/embeddings', [
            'input' => $text,
            'model' => $this->model,
            'encoding_format' => 'float',
        ]);

        if (!$response->successful()) {
            Log::error('Embedding generation failed', [
                'status' => $response->status(),
                'body' => $response->body(),
            ]);
            throw new \RuntimeException('Failed to generate embedding: ' . $response->body());
        }

        return $response->json('data.0.embedding');
    }

    public function generateBatch(array $texts): array
    {
        // OpenAI supports batching up to 2048 inputs
        $chunks = array_chunk($texts, 100);
        $allEmbeddings = [];

        foreach ($chunks as $batch) {
            $response = Http::withHeaders([
                'Authorization' => 'Bearer ' . $this->apiKey,
                'Content-Type' => 'application/json',
            ])->post('https://api.openai.com/v1/embeddings', [
                'input' => $batch,
                'model' => $this->model,
            ]);

            foreach ($response->json('data') as $item) {
                $allEmbeddings[] = $item['embedding'];
            }

            // Rate limit consideration
            usleep(100000); // 100ms between batches
        }

        return $allEmbeddings;
    }

    private function truncateToTokenLimit(string $text, int $maxTokens): string
    {
        // Approximation: 1 token ≈ 4 characters for English
        $maxChars = $maxTokens * 4;
        return substr($text, 0, $maxChars);
    }
}

The Core Ingestion Job

Wire everything together in a queueable job for background processing:

<?php

namespace App\Jobs;

use App\Models\Document;
use App\Services\ChunkingService;
use App\Services\EmbeddingService;
use App\Services\PdfExtractionService;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\DB;

class ProcessDocumentJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public int $tries = 3;
    public int $timeout = 300;

    public function __construct(
        private Document $document
    ) {}

    public function handle(
        PdfExtractionService $extractor,
        ChunkingService $chunker,
        EmbeddingService $embedder
    ): void {
        $extracted = $extractor->extract($this->document->file_path);
        
        $this->document->update([
            'title' => $extracted['title'],
            'total_pages' => $extracted['total_pages'],
        ]);

        $chunks = $chunker->chunk($extracted['pages']);
        
        // Batch generate embeddings for efficiency
        $texts = array_column($chunks, 'content');
        $embeddings = $embedder->generateBatch($texts);

        DB::transaction(function () use ($chunks, $embeddings) {
            foreach ($chunks as $index => $chunk) {
                \App\Models\DocumentChunk::create([
                    'document_id' => $this->document->id,
                    'chunk_index' => $chunk['chunk_index'],
                    'content' => $chunk['content'],
                    'page_number' => $chunk['page_number'],
                    'embedding' => $embeddings[$index],
                ]);
            }
        });
    }
}

Similarity Search with pgvector

Now for the retrieval part. pgvector supports multiple distance operators. For cosine similarity, which OpenAI embeddings are normalized for, use

<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Relations\BelongsTo;

class DocumentChunk extends Model
{
    use HasFactory;

    protected $fillable = [
        'document_id',
        'chunk_index',
        'content',
        'page_number',
        'embedding',
    ];

    protected $casts = [
        'embedding' => 'vector',
    ];

    public function document(): BelongsTo
    {
        return $this->belongsTo(Document::class);
    }

    public function scopeSimilarTo($query, array $embedding, float $threshold = 0.7)
    {
        return $query
            ->selectRaw('
                document_chunks.*,
                1 - (embedding <=> ?) as similarity
            ', [json_encode($embedding)])
            ->whereRaw('
                1 - (embedding <=> ?) > ?
            ', [json_encode($embedding), $threshold])
            ->orderByRaw('embedding <=> ?', [json_encode($embedding)]);
    }
}

The <=> operator computes cosine distance (1 - cosine similarity). We convert this to similarity score for threshold filtering and ranking.

Building the RAG Controller

Expose retrieval through a clean API endpoint:

<?php

namespace App\Http\Controllers\Api;

use App\Http\Controllers\Controller;
use App\Models\DocumentChunk;
use App\Services\EmbeddingService;
use Illuminate\Http\JsonResponse;
use Illuminate\Http\Request;

class RagController extends Controller
{
    public function __construct(
        private EmbeddingService $embedder
    ) {}

    public function retrieve(Request $request): JsonResponse
    {
        $request->validate([
            'query' => 'required|string|max:1000',
            'top_k' => 'integer|min:1|max:20',
            'threshold' => 'numeric|min:0|max:1',
        ]);

        $queryEmbedding = $this->embedder->generate($request->input('query'));
        
        $results = DocumentChunk::with('document')
            ->similarTo(
                $queryEmbedding,
                $request->input('threshold', 0.7)
            )
            ->limit($request->input('top_k', 5))
            ->get();

        return response()->json([
            'query' => $request->input('query'),
            'results' => $results->map(function ($chunk) {
                return [
                    'content' => $chunk->content,
                    'similarity' => (float) $chunk->similarity,
                    'source' => [
                        'document' => $chunk->document->title,
                        'page' => $chunk->page_number,
                    ],
                ];
            }),
        ]);
    }
}

Integration with LLM Completion

Your frontend or another service takes these retrieved chunks and builds the context for an LLM:

function buildRagPrompt(string $userQuery, array $retrievedChunks): string
{
    $context = implode("\n\n---\n\n", array_map(
        fn($c) => "Source: {$c['source']['document']} (Page {$c['source']['page']})\n{$c['content']}",
        $retrievedChunks
    ));

    return <<<PROMPT
You are a helpful assistant answering questions based on internal company documents.

Use ONLY the following retrieved context to answer. If the context doesn't contain the answer, say "I don't have sufficient information."

Retrieved Context:
{$context}

User Question: {$userQuery}

Answer:
PROMPT;
}

Production Considerations

Index your vectors for performance at scale:

CREATE INDEX ON document_chunks USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);

Choose lists based on row count—roughly square root of total chunks. For millions of chunks, consider hnsw indexes instead:

CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

Monitor embedding costs. At $0.02 per 1M tokens for text-embedding-3-small, a 100-page technical manual might cost $0.30-0.50 to embed. Budget for re-embedding when documents update.

Handle updates gracefully. When a PDF version changes, delete old chunks before inserting new ones, or implement soft deletes with versioning.

Conclusion

This pipeline gives you production-ready RAG infrastructure using tools your team already knows. Laravel's queue system handles backpressure during bulk imports. PostgreSQL with pgvector keeps your data in one place with ACID guarantees. The modular service design lets you swap PDF extractors, embedding providers, or chunking strategies without rewriting everything.

Start with this foundation, measure your retrieval quality with real queries, and iterate on chunk size and overlap based on your document types. The best RAG system is one you can observe and improve over time.