examples/sum/hardware/generate-recordbatch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate a RecordBatch for simulation\n",
    "\n",
    "In this step, we will generate a tiny RecordBatch that we can use to simulate our FPGA kernel.\n",
    "We will start by reading the schema created in [the previous step](generate-schema.ipynb) from a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyarrow as pa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number: int64 not null\n",
      "metadata\n",
      "--------\n",
      "OrderedDict([(b'fletcher_mode', b'read'), (b'fletcher_name', b'ExampleBatch')])\n"
     ]
    }
   ],
   "source": [
    "# Open the file created in the previous example as an input stream.\n",
    "stream = pa.input_stream('schema.as')\n",
    "\n",
    "# Read the Arrow schema from the stream.\n",
    "schema = pa.read_schema(stream)\n",
    "\n",
    "# Show contents.\n",
    "print(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, we've retreived the original schema with the `number` field. \n",
    "\n",
    "Now we will create an actual dataset based on this schema. It will be a tiny dataset, just for demonstration purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "  1,\n",
      "  -3,\n",
      "  3,\n",
      "  -7\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "# Create a list of PyArrow Arrays. Every Array can be seen \n",
    "# as a 'Column' of the RecordBatch we will create.\n",
    "data = [pa.array([1, -3, 3, -7])]\n",
    "\n",
    "# Create a RecordBatch from the Arrays.\n",
    "recordbatch = pa.RecordBatch.from_arrays(data, schema)\n",
    "\n",
    "# Print the first (and only) column of this RecordBatch.\n",
    "print(recordbatch.column(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've created a little RecordBatch, we would like to save it as a file that we can give to Fletchgen. Fletchgen can take the RecordBatch and generate a simulation top-level. It can then initialize simulation memory contents with the RecordBatch. You can use this to verify the functionality of your kernel in simulation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an Arrow RecordBatchFileWriter.\n",
    "writer = pa.RecordBatchFileWriter('recordbatch.rb', schema)\n",
    "\n",
    "# Write the RecordBatch.\n",
    "writer.write(recordbatch)\n",
    "\n",
    "# Close the writer.\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have now created a small RecordBatch for simulation purposes and saved it as a file, so we can pass it to Fletchgen."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}