WhizzML Resources¶
WhizzML is a Domain Specific Language that allows the definition and
execution of ML-centric workflows. Its objective is allowing BigML
users to define their own composite tasks, using as building blocks
the basic resources provided by BigML itself. Using Whizzml they can be
glued together using a higher order, functional, Turing-complete language.
The WhizzML code can be stored and executed in BigML using three kinds of
resources: Scripts
, Libraries
and Executions
.
WhizzML Scripts
can be executed in BigML’s servers, that is,
in a controlled, fully-scalable environment which takes care of their
parallelization and fail-safe operation. Each execution uses an Execution
resource to store the arguments and results of the process. WhizzML
Libraries
store generic code to be shared of reused in other WhizzML
Scripts
.
Scripts¶
In BigML a Script
resource stores WhizzML source code, and the results of
its compilation. Once a WhizzML script is created, it’s automatically compiled;
if compilation succeeds, the script can be run, that is,
used as the input for a WhizzML execution resource.
An example of a script
that would create a source
in BigML using the
contents of a remote file is:
>>> from bigml.api import BigML
>>> api = BigML()
# creating a script directly from the source code. This script creates
# a source uploading data from an s3 repo. You could also create a
# a script by using as first argument the path to a .whizzml file which
# contains your source code.
>>> script = api.create_script( \
"(create-source {\"remote\" \"s3://bigml-public/csv/iris.csv\"})")
>>> api.ok(script) # waiting for the script compilation to finish
>>> api.pprint(script['object'])
{ u'approval_status': 0,
u'category': 0,
u'code': 200,
u'created': u'2016-05-18T16:54:05.666000',
u'description': u'',
u'imports': [],
u'inputs': None,
u'line_count': 1,
u'locale': u'en-US',
u'name': u'Script',
u'number_of_executions': 0,
u'outputs': None,
u'price': 0.0,
u'private': True,
u'project': None,
u'provider': None,
u'resource': u'script/573c9e2db85eee23cd000489',
u'shared': False,
u'size': 59,
u'source_code': u'(create-source {"remote" "s3://bigml-public/csv/iris.csv"})',
u'status': { u'code': 5,
u'elapsed': 4,
u'message': u'The script has been created',
u'progress': 1.0},
u'subscription': True,
u'tags': [],
u'updated': u'2016-05-18T16:54:05.850000',
u'white_box': False}
A script
allows to define some variables as inputs
. In the previous
example, no input has been defined, but we could modify our code to
allow the user to set the remote file name as input:
>>> from bigml.api import BigML
>>> api = BigML()
>>> script = api.create_script( \
"(create-source {\"remote\" my_remote_data})",
{"inputs": [{"name": "my_remote_data",
"type": "string",
"default": "s3://bigml-public/csv/iris.csv",
"description": "Location of the remote data"}]})
The script
can also use a library
resource (please, see the
Libraries
section below for more details) by including its id in the
imports
attribute. Other attributes can be checked at the
API Developers documentation for Scripts.
Executions¶
To execute in BigML a compiled WhizzML script
you need to create an
execution
resource. It’s also possible to execute a pipeline of
many compiled scripts in one request.
Each execution
is run under its associated user credentials and its
particular environment constrains. As scripts
can be shared,
different users can execute the same script
using different inputs.
Each particular execution will generate an execution
resource in BigML.
As an example of an execution
resource, let’s create one for the first
script in the previous section. In this case, no inputs are required because
the script
expects none:
>>> from bigml.api import BigML
>>> api = BigML()
>>> execution = api.create_execution('script/573c9e2db85eee23cd000489')
>>> api.ok(execution) # waiting for the execution to finish
>>> api.pprint(execution['object'])
{ u'category': 0,
u'code': 200,
u'created': u'2016-05-18T16:58:01.613000',
u'creation_defaults': { },
u'description': u'',
u'execution': { u'output_resources': [ { u'code': 1,
u'id': u'source/573c9f19b85eee23c600024a',
u'last_update': 1463590681854,
u'progress': 0.0,
u'state': u'queued',
u'task': u'Queuing job',
u'variable': u''}],
u'outputs': [],
u'result': u'source/573c9f19b85eee23c600024a',
u'results': [u'source/573c9f19b85eee23c600024a'],
u'sources': [[ u'script/573c9e2db85eee23cd000489',
u'']],
u'steps': 16},
u'inputs': None,
u'locale': u'en-US',
u'name': u"Script's Execution",
u'project': None,
u'resource': u'execution/573c9f19b85eee23bd000125',
u'script': u'script/573c9e2db85eee23cd000489',
u'script_status': True,
u'shared': False,
u'status': { u'code': 5,
u'elapsed': 249,
u'elapsed_times': { u'in-progress': 247,
u'queued': 62,
u'started': 2},
u'message': u'The execution has been created',
u'progress': 1.0},
u'subscription': True,
u'tags': [],
u'updated': u'2016-05-18T16:58:02.035000'}
As you can see, the execution resource contains information about the result of the execution, the resources that have been generated while executing and users can define some variables in the code to be exported as outputs.
An execution
receives inputs, the ones defined in the script
chosen
to be executed, and generates a result. It can also generate outputs and
create resources. To
execute a script
that expects some inputs, you will need to specify the
concrete values of those inputs, unless a default value has been assigned
for them in the script’s inputs definition. Following the second example in
the previous section, we can execute the script that creates a source from a
URL pointing to a CSV file:
>>> from bigml.api import BigML
>>> api = BigML()
>>> execution = api.create_execution( \
script,
{"inputs": [["my_remote_data",
"https://static.bigml.com/csv/iris.csv"]]})
For more details on executions’ structure, please refer to the Developers documentation for Executions.
The results of an execution can be easily obtained by using the Execution
class. This class can be used to instantiate a local object that will
expose the result, outputs and output resources generated in the execution
in its attributes.
from bigml.execution import Execution
execution = Execution("execution/5cae5ad4b72c6609d9000468")
print "The result of the execution is %s" % execution.result
print " and the output for variable 'my_variable': %s" % \
execution.outputs["my_variable"]
print "The resources created in the execution are: %s" % \
execution.output_resources
As an execution is in progress, the execution.result
attribute will
contain the value of the last evaluated expression at that point.
Therefore, the value of the result
attribute will change untill it
will contain the final result of the execution when finished.
Also, if the execution fails, the error information can be found in the corresponding attributes:
from bigml.execution import Execution
execution = Execution("execution/5cae5ad4b72c6609d9000468")
print "The status of the execution is %s" % execution.status
print "The execution failed at %s with error %s: %s" % ( \
execution.error_location, execution.error, execution.error_message)
Libraries¶
The library
resource in BigML stores a special kind of compiled Whizzml
source code that only defines functions and constants. The library
is
intended as an import for executable scripts.
Thus, a compiled library cannot be executed, just used as an
import in other libraries
and scripts
(which then have access
to all identifiers defined in the library
).
As an example, we build a library
to store the definition of two functions:
mu
and g
. The first one adds one to the value set as argument and
the second one adds two variables and increments the result by one.
>>> from bigml.api import BigML
>>> api = BigML()
>>> library = api.create_library( \
"(define (mu x) (+ x 1)) (define (g z y) (mu (+ y z)))")
>>> api.ok(library) # waiting for the library compilation to finish
>>> api.pprint(library['object'])
{ u'approval_status': 0,
u'category': 0,
u'code': 200,
u'created': u'2016-05-18T18:58:50.838000',
u'description': u'',
u'exports': [ { u'name': u'mu', u'signature': [u'x']},
{ u'name': u'g', u'signature': [u'z', u'y']}],
u'imports': [],
u'line_count': 1,
u'name': u'Library',
u'price': 0.0,
u'private': True,
u'project': None,
u'provider': None,
u'resource': u'library/573cbb6ab85eee23c300018e',
u'shared': False,
u'size': 53,
u'source_code': u'(define (mu x) (+ x 1)) (define (g z y) (mu (+ y z)))',
u'status': { u'code': 5,
u'elapsed': 2,
u'message': u'The library has been created',
u'progress': 1.0},
u'subscription': True,
u'tags': [],
u'updated': u'2016-05-18T18:58:52.432000',
u'white_box': False}
Libraries can be imported in scripts. The imports
attribute of a script
can contain a list of library
IDs whose defined functions
and constants will be ready to be used throughout the script
. Please,
refer to the API Developers documentation for Libraries
for more details.