Batch Geocoding – Technical Details

Back to Demo

High-Level System Overview

  • Purpose: Bulk geocode CSV address files into match status, matched address, score, latitude, and longitude fields using ArcGIS findAddressCandidates.
  • Business problem: Converts manual address cleanup into a repeatable API workflow that can process duplicate-heavy datasets without issuing unnecessary upstream geocoding calls.
  • Architecture: Razor Page upload experience → versioned ASP.NET Core API → IBatchGeocodingService → channel-based worker pipeline → ArcGIS REST API, with job state stored behind IBatchJobStore.
  • Technologies: .NET 10, Razor Pages, vanilla JavaScript, Bootstrap, DataTables.js, typed HttpClient, Polly resilience, IDistributedCache, Redis or in-memory fallback.
  • Deployment model: Runs as part of the Portfolio web application on Render, supports Docker Compose with Redis locally, and has Kubernetes manifests for horizontally scaled replicas.

Frontend Architecture

The UI is a Razor Page backed by wwwroot/js/BatchGeocoding/app.js. It keeps state in page-level JavaScript rather than a SPA framework: selected file, submitted job id, polling state, progress metrics, and result rows are managed in the browser and reflected into Bootstrap cards and a DataTables grid.

  • API communication: Shared route constants from api-config.js point to /api/v1/geocoding/batch; shared fetch helpers send cookies and normalize error handling.
  • Upload flow: The browser posts multipart/form-data without manually setting Content-Type, allowing the browser to generate the boundary correctly.
  • Async UX: The recommended path returns 202 Accepted with a status URL; the client polls until the job reaches a terminal state, then renders metrics and row-level results.
  • Performance: DataTables provides client-side filtering, sorting, and export for completed results, keeping the server focused on geocoding and job orchestration.

API Layer

  • Controller: BatchGeocodingController is a versioned API controller at /api/v1/geocoding/batch and is intentionally [AllowAnonymous].
  • Endpoints: POST /api/v1/geocoding/batch enqueues a job; GET /api/v1/geocoding/batch/{jobId}/status returns progress and results; POST /api/v1/geocoding/batch/sync is retained as an obsolete compatibility path.
  • Validation: The controller rejects null or empty files before calling the service; deeper CSV validation occurs in the service.
  • Error handling: ArgumentException maps to 400, an open Polly circuit maps to 503 with retry guidance, and unexpected errors flow through the global exception middleware.
  • Security: No user identity is required; the API accepts only uploaded files and does not trust caller-supplied user ownership fields.

Services and Business Logic

  • Service boundary: BatchGeocodingService owns CSV parsing, job creation, pipeline orchestration, ArcGIS calls, cache lookup, match scoring, and DTO mapping.
  • Background processing: The request thread reads the uploaded file into memory before background work starts, avoiding lifetime issues with IFormFile streams.
  • Concurrency model: A bounded System.Threading.Channels producer/consumer pipeline applies back-pressure and limits ArcGIS concurrency with BatchGeocoding:MaxConcurrency.
  • Resilience: Typed HttpClient is wrapped with timeout, exponential retry with jitter, and circuit breaker policies for transient ArcGIS failures.
  • Match logic: Candidates below BatchGeocoding:MinMatchScore are returned as unmatched rather than hidden, preserving auditability of failed rows.

Data Access Layer

Batch geocoding is primarily external-service and cache driven. It does not persist each geocoded row to PostgreSQL; instead, job snapshots are stored through IBatchJobStore and result cache entries use IDistributedCache.

  • Job state: RedisBatchJobStore stores the full BatchJob JSON document with a 24-hour TTL when Redis is configured; InMemoryBatchJobStore supports local development.
  • Caching: Normalized address strings key Redis-backed IDistributedCache entries so repeated addresses across a batch can avoid duplicate ArcGIS calls; local development falls back to MemoryDistributedCache.
  • Serialization: Shared PortfolioJsonOptions.Default keeps enum and DTO serialization consistent across Redis and API responses.
  • Scalability: Redis-backed job state lets any replica serve status polling requests after a different replica accepted the upload, while shared cache entries reduce duplicate upstream work across replicas.

Infrastructure and Deployment

  • Current hosting: The application is deployed on Render with configuration injected through environment variables.
  • Containerization: The root Dockerfile builds a multi-stage .NET image; Docker Compose runs the app with a Redis sidecar for local distributed-cache testing.
  • Kubernetes readiness: Manifests define a multi-replica deployment, Redis service for shared geocoding cache, batch job state, and Data Protection keys, plus resource requests/limits, service, and HPA.
  • Configuration: BatchGeocoding:MaxConcurrency, MinMatchScore, CacheTtlMinutes, and Redis__ConnectionString are environment-configurable.
  • Cloud portability: The design can move to Azure App Service, Azure Container Apps, or AKS with Azure Cache for Redis and managed PostgreSQL without changing service contracts.

Engineering Decisions and Tradeoffs

  • Channels over unbounded tasks: Bounded channels make throughput predictable and prevent large uploads from creating unbounded concurrent ArcGIS calls.
  • Async job over synchronous request: The job pattern avoids long-running HTTP requests and supports scale-out status polling, at the cost of added job-state complexity.
  • Cache before persistence: Geocoding outputs are treated as transient workflow results; persisting only job snapshots keeps the database clean but limits historical analytics.
  • Future improvements: Add durable queue-backed workers, per-client rate limits, file size quotas, idempotency keys, dead-letter handling, and managed Redis/PostgreSQL in production cloud environments.

Interview Discussion Points

  • How would you set concurrency limits against a third-party geocoding API with variable latency and quotas?
  • Why is Redis-backed job state important once the app runs more than one replica?
  • How would you evolve this from in-process background work to a queue/worker architecture?
  • Where would you add rate limiting, file validation, observability, and tenant-level quotas?
  • How do timeout, retry, and circuit breaker policies interact with user-facing SLA and upstream protection?