What is FileDrop?

FileDrop on AWS is Observe's secure and serverless way to ingest files from any source without having to grant access to your AWS environment. You can ingest data from AWS S3 directly into Observe's Snowflake backend via Snowpipe, bypassing the traditional HTTP ingest pipeline and offering the following benefits:

• Large file and format-agnostic support: S3 can handle individual files up to files up to 150MB in size, with a limit of 5TB for any single object. You can drop any file type including binary, text, images, and executables. Note that it's common for administrators to use S3 bucket policies to block executables files from being uploaded per security guidelines. File names must be UTF-8 encoded.
• Monitoring: Set up monitoring for FileDrop failures, along with automated alerts and notifications. To check for failures, you can monitor metrics such as OnUploadExecutionsFailed in Amazon CloudWatch, or investigate Amazon CloudWatch logs for errors. See AWS Transfer Family managed workflows for more information.
• Security and compliance: Files are encrypted at rest and in transit. You can also ensure data stays in a specific geographic region for legal reasons (GDPR/CCPA).
• Automation: As soon as a file is dropped, AWS can automatically trigger a workflow to process the data, notify a team via Slack or email, or start a database import.
• Zero Infrastructure: Because it’s serverless, you don't have to manage or patch any servers; you only pay when someone actually uploads a file.

The following flowchart shows the FileDrop process and workflow:

flowchart LR
    subgraph Customer["Customer's AWS Account"]
        A[Amazon S3<br/>Customer bucket]
    end

    subgraph Observe["Observe / Snowpipe Auto Ingest"]
        B[Amazon S3<br/>Observe-managed bucket]
        C[Amazon SQS]
        D[Snowpipe]
    end

    subgraph Snowflake["Snowflake"]
        E[(Snowflake Table)]
    end

    T[Transloader]

    %% Main ingest flow
    A --> B
    B --> C
    C --> D
    D --> E

    %% Transloader path
    T --> E