x/stacktach
Code Issues Proposed changes
stacktach/docs/dbapi.rst
Manali Latkar cfab4cc6cc api to store AH event id along with send_status by yagi 
Change-Id: Ied32cb88e5fbcb000c5b32f4d1190da6da3dc8ec

basic debianization

Change-Id: I1fdcb4c7040f350e034e2cfd44272a7b35178221

basic debianization

Change-Id: I8ba97fc088426b07f583c4335a695ec7a7af49b4

Making the type and null check for os_version optional according to import image_type

Change-Id: Ide512b4462d5c2d21ec1538325e5c2971b6b2934

Adding host and deployment info to missing exists entries in the nova usage audit

Change-Id: Icc1d093ab42e8164dfe98133458ede056becfaa0

Added column headers for host and deployment in json reports

Change-Id: Ic8a7b171b4f717e1a2da2e626eb5bcf1863037e1

Switched to Apache licensing

Change-Id: Ifdbc6c46d51913dacb8fca4a8f770f3f6f57f8f8

Freshen up with latest from RackerLabs (and include tox.ini)

Added instance hours report

Initial version of report to calculate unit hours used
for nova instances

Breakdown by flavor, flavor class, account/billing types and by tenant.

Moved license so script has shebang as the first line
Add tenant info cache.
Refactor Instance hr report.
Added cache table for basic tenant info for reports.
Refactor instance_hours report to use table.
Improve performance of tenant info update.

use bulk sql operations to speed up the tenant info update,
as it's taking ~40s/1000 tenants to update on a decent machine.

Fix some tests broken by rebase. Fix unittests broken by
rebase. Also, renumber migration due to collision.

Add Apache license header to new files.

Fixed bug with fetching deployment information in
reconciler. Reverted old method for fetching
current usage's deployment and added new method to
fetch latest deployment information for
a request_id.

Made the field mismatch error message more readable
Refactored nova and glance verifier tests

the exists are updated with 201 send_status as part of stacktach down repair mechanism

Revert "Fixed bug with fetching deployment information in"

Revert "Adding host and deployment info to missing exists entries in the nova usage audit"

Revert "Added column headers for host and deployment in json reports"

Only log ERROR on last retry

fixed the wrong status name for sent_failed variable in audit report

fixing documentation for urls that are not available for glance

deprecating stacky urls (usage, deletes, exists) that are not
used anymore

Revert "Revert "Added column headers for host and deployment in json reports""

Revert "Revert "Adding host and deployment info to missing exists entries in the nova usage audit""

Revert "Revert "Fixed bug with fetching deployment information in""

Cell and compute info added for verification failures as well.
If that is not present(request_id is not populated for an
InstanceUsage entry), the cells display '-'

Add tox support for move to stackforge

Add tox support for move to stackforge

Change-Id: Id94c2a7f1f9061e972e90c3f54e39c9dec11943b
2014-07-03 20:07:24 +05:30

25 KiB
Raw Blame History

The StackTach Database REST Interface ############################

JSON Response Format

The StackTach Database API uses a more standard data model for access to database objects. The Database API is read only, with the exception of usage confirmation, which is used to indicate that usage has been sent downstream.

The JSON response format uses an envelope with a single key to indicate the type of object returned. This object can be either a dictionary in the case of queries that return single objects, or a list when multiple objects are turned.

Sample JSON response, single object: :

{
  "enitity":
  {
    "id": 1
    "key1": "value1",
    "key2": "value2"
  }
}

Sample JSON response, multiple objects: :

{
  "enitities":
  [
    {
      "id": 1,
      "key1": "value1",
      "key2": "value2"
    },
    {
      "id": 2,
      "key1": "value1",
      "key2": "value2"
    }
  ]
}

Write APIs

db/confirm/usage/exists/batch/

Uses the provided message_id's and http status codes to update image and instance exists send_status values.

Example V0 request:

PUT db/confirm/usage/exists/batch/ HTTP/1.1
Host: example.com
Accept: application/json

{
  "messages":
  [
    {"nova_message_id": 200},
    {"nova_message_id": 400}
  ]
}

Example V1 request:

PUT db/confirm/usage/exists/batch/ HTTP/1.1
Host: example.com
Accept: application/json

{
  "messages":
  [
    {
      "nova":
      [
        {"nova_message_id1": 200},
        {"nova_message_id2": 400}
      ],
      "glance":
      [
        {"glance_message_id1": 200},
        {"glance_message_id2": 400}
      ]
    }
  ]
  "version": 1
}

Example V2 request:

PUT db/confirm/usage/exists/batch/ HTTP/1.1
Host: example.com
Accept: application/json

{
  "messages":
  [
    {
      "nova":
      [
        {"nova_message_id1":
            {
              "event_id": "AH_event_id_1",
              "status": 201
            }
        },
        {"nova_message_id2":
            {
              "event_id": "AH_event_id_2",
              "status": 201
            }
        }
      ],
      "glance":
      [
        {"glance_message_id1":
             {
              "event_id": "AH_event_id_3",
              "status": 201},
              }
        },
        {"glance_message_id2":
             {
              "event_id": "AH_event_id_3",
              "status": 201
              }
        }
      ]
    }
  ]
  "version": 2
}

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

Read APIs

db/stats/events

Returns a count of events stored in Stacktach's Rawdata tables from when_min to when_max

Query Parameters

  • event: event type to filter by
  • when_min: datetime (yyyy-mm-dd hh:mm:ss)
  • when_max: datetime (yyyy-mm-dd hh:mm:ss)
  • service: nova or glance. default="nova"

Example request:

GET db/stats/events/ HTTP/1.1
Host: example.com
Accept: application/json

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

{
  count: 10
}

db/stats/nova/exists/

Returns a list of status combinations and count of events with those status combinations.

Note: Only status combinations with >0 count will show up.

Query Parameters

  • audit_period_beginning_min: datetime (yyyy-mm-dd hh:mm:ss)
  • audit_period_beginning_max: datetime (yyyy-mm-dd hh:mm:ss)
  • audit_period_ending_min: datetime (yyyy-mm-dd hh:mm:ss)
  • audit_period_ending_max: datetime (yyyy-mm-dd hh:mm:ss)
  • launched_at_min: datetime (yyyy-mm-dd hh:mm:ss)
  • launched_at_max: datetime (yyyy-mm-dd hh:mm:ss)
  • deleted_at_min: datetime (yyyy-mm-dd hh:mm:ss)
  • deleted_at_max: datetime (yyyy-mm-dd hh:mm:ss)
  • received_min: datetime (yyyy-mm-dd hh:mm:ss)
  • received_max: datetime (yyyy-mm-dd hh:mm:ss)

Example request:

GET /db/stats/nova/exists/ HTTP/1.1
Host: example.com
Accept: application/json

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

{
  "stats":
  [
    {"status": "pending", "send_status": 0, "event_count": 1},
    {"status": "verified", "send_status": 200, "event_count": 100},
    {"status": "reconciled", "send_status": 200, "event_count": 2},
    {"status": "failed", "send_status": 0, "event_count": 1},
  ]
}

db/stats/glance/exists/

Returns a list of status combinations and count of events with those status combinations.

Note: Only status combinations with >0 count will show up.

Query Parameters

  • audit_period_beginning_min: datetime (yyyy-mm-dd hh:mm:ss)
  • audit_period_beginning_max: datetime (yyyy-mm-dd hh:mm:ss)
  • audit_period_ending_min: datetime (yyyy-mm-dd hh:mm:ss)
  • audit_period_ending_max: datetime (yyyy-mm-dd hh:mm:ss)
  • created_at_min: datetime (yyyy-mm-dd hh:mm:ss)
  • created_at_max: datetime (yyyy-mm-dd hh:mm:ss)
  • deleted_at_min: datetime (yyyy-mm-dd hh:mm:ss)
  • deleted_at_max: datetime (yyyy-mm-dd hh:mm:ss)
  • received_min: datetime (yyyy-mm-dd hh:mm:ss)
  • received_max: datetime (yyyy-mm-dd hh:mm:ss)

Example request:

GET /db/stats/nova/exists/ HTTP/1.1
Host: example.com
Accept: application/json

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

{
  "stats":
  [
    {"status": "verified", "send_status": 200, "event_count": 200},
    {"status": "failed", "send_status": 0, "event_count": 2},
  ]
}

db/usage/launches/

Deprecated, see: dbapi-nova-launches

db/usage/nova/launches/

Returns a list of instance launches matching provided query criteria.

Query Parameters

  • launched_at_min: datetime (yyyy-mm-dd hh:mm:ss)
  • launched_at_max: datetime (yyyy-mm-dd hh:mm:ss)
  • instance: uuid
  • limit: int, default: 50, max: 1000
  • offset: int, default: 0

Example request:

GET /db/usage/nova/launches/ HTTP/1.1
Host: example.com
Accept: application/json

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

{
  "launches":
  [
    {
      "os_distro": "org.centos",
      "os_version": "5.8",
      "instance_flavor_id": "2",
      "instance_type_id": "2",
      "launched_at": "2014-01-17 15:35:44",
      "instance": "72e4d8e8-9f63-47cb-a904-0193e5edac6e",
      "os_architecture": "x64",
      "request_id": "req-7a86ed49-e1f4-4403-b3ef-22636f7acb7d",
      "rax_options": "0",
      "id": 91899,
      "tenant": "5853600"
    },
    {
      "os_distro": "org.centos",
      "os_version": "5.8",
      "instance_flavor_id": "performance1-4",
      "instance_type_id": "11",
      "launched_at": "2014-01-17 15:35:20",
      "instance": "932bcfd9-af68-4261-805e-6e43156c3b40",
      "os_architecture": "x64",
      "request_id": "req-6bfe911f-40f2-4fd8-946a-070c10bed014",
      "rax_options": "0",
      "id": 91898,
      "tenant": "5853595"
    }
  ]
}

db/usage/glance/images/

Returns a list of images matching provided query criteria.

Query Parameters

  • created_at_min: datetime (yyyy-mm-dd hh:mm:ss)
  • created_at_max: datetime (yyyy-mm-dd hh:mm:ss)
  • limit: int, default: 50, max: 1000
  • offset: int, default: 0

Example request:

GET /db/usage/glance/images/ HTTP/1.1
Host: example.com
Accept: application/json

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

{
  "images":
  [
    {
      "uuid": "2048efd8-fdce-4123-bdbc-add3bfe64b83",
      "created_at": "2014-01-17 02:28:08",
      "owner": null,
      "last_raw": 299977,
      "id": 4837,
      "size": 9192352
    },
    {
      "uuid": "aa2c07dd-fd1c-4ad3-9f73-6a6d7d8a0dbd",
      "created_at": "2014-01-17 02:24:18",
      "owner": "5937488",
      "last_raw": 299967,
      "id": 4836,
      "size": 9
    }
  ]
}

db/usage/launches/<launch_id>/

Deprecated, see: dbapi-nova-launch

db/usage/nova/launches/<launch_id>/

Returns the single launch with id matching the provided id.

Example request:

GET /db/usage/nova/launches/91898/ HTTP/1.1
Host: example.com
Accept: application/json

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

{
  "launch":
  {
    "os_distro": "org.centos",
    "os_version": "5.8",
    "instance_flavor_id": "performance1-4",
    "instance_type_id": "11",
    "launched_at": "2014-01-17 15:35:20",
    "instance": "932bcfd9-af68-4261-805e-6e43156c3b40",
    "os_architecture": "x64",
    "request_id": "req-6bfe911f-40f2-4fd8-946a-070c10bed014",
    "rax_options": "0",
    "id": 91898,
    "tenant": "5853595"
  }
}

db/usage/glance/images/<image_id>/

Returns the single image with id matching the provided id.

Example request:

GET /db/usage/glance/images/4836/ HTTP/1.1
Host: example.com
Accept: application/json

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

{
  "launch":
  {
    "uuid": "aa2c07dd-fd1c-4ad3-9f73-6a6d7d8a0dbd",
    "created_at": "2014-01-17 02:24:18",
    "owner": "5937488",
    "last_raw": 299967,
    "id": 4836,
    "size": 9
  }
}

db/usage/deletes/

Deprecated, see: dbapi-nova-deletes

db/usage/nova/deletes/

Returns a list of instance deletes matching provided query criteria.

Query Parameters

  • launched_at_min: datetime (yyyy-mm-dd hh:mm:ss)
  • launched_at_max: datetime (yyyy-mm-dd hh:mm:ss)
  • deleted_at_min: datetime (yyyy-mm-dd hh:mm:ss)
  • deleted_at_max: datetime (yyyy-mm-dd hh:mm:ss)
  • instance: uuid
  • limit: int, default: 50, max: 1000
  • offset: int, default: 0

Example request:

GET /db/usage/nova/deletes/ HTTP/1.1
Host: example.com
Accept: application/json

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

{
  "deletes":
  [
    {
      "raw": 14615347,
      "instance": "b36a8c2d-af88-4371-b14c-14dadf7073e5",
      "deleted_at": "2014-01-17 16:07:30",
      "id": 65110,
      "launched_at": "2014-01-17 16:06:54"
    },
    {
      "raw": 14615248,
      "instance": "3fd6797d-bc35-42d9-ad85-157a2ea93023",
      "deleted_at": "2014-01-17 16:05:23",
      "id": 65108,
      "launched_at": "2014-01-17 16:05:00"
    }
  ]
}

db/usage/glance/deletes/

Returns a list of image deletes matching provided query criteria.

Query Parameters

  • deleted_at_min: datetime (yyyy-mm-dd hh:mm:ss)
  • deleted_at_max: datetime (yyyy-mm-dd hh:mm:ss)
  • limit: int, default: 50, max: 1000
  • offset: int, default: 0

Example request:

GET /db/usage/glance/deletes/ HTTP/1.1
Host: example.com
Accept: application/json

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

{
  "deletes":
  [
    {
      "raw": 300523,
      "deleted_at": "2014-01-17 15:28:18.154927",
      "id": 3169,
      "uuid": "f8b02f0e-b392-40f5-9d39-0458ae6ebfb3"
    },
    {
      "raw": 300512,
      "deleted_at": "2014-01-17 14:28:20.544617",
      "id": 3168,
      "uuid": "4c9dc0be-856b-4e98-81a5-1b63df108e7d"
    }
  ]
}

db/usage/deletes/<delete_id>/

Deprecated, see: dbapi-nova-delete

db/usage/nova/deletes/<delete_id>/

Returns the single instance delete with id matching the provided id.

Example request:

GET /db/usage/nova/deletes/65110/ HTTP/1.1
Host: example.com
Accept: application/json

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

{
  "delete":
  {
    "raw": 14615347,
    "instance": "b36a8c2d-af88-4371-b14c-14dadf7073e5",
    "deleted_at": "2014-01-17 16:07:30",
    "id": 65110,
    "launched_at": "2014-01-17 16:06:54"
  }
}

db/usage/glance/deletes/<delete_id>/

Returns the single image delete with id matching the provided id.

Example request:

GET /db/usage/glance/deletes/3168/ HTTP/1.1
Host: example.com
Accept: application/json

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

{
  "delete":
  {
    "raw": 300512,
    "deleted_at": "2014-01-17 14:28:20.544617",
    "id": 3168,
    "uuid": "4c9dc0be-856b-4e98-81a5-1b63df108e7d"
  }
}

db/usage/exists/

Deprecated, see: dbapi-nova-exists

db/usage/nova/exists/

Returns a list of instance exists matching provided query criteria.

Query Parameters

  • audit_period_beginning_min: datetime (yyyy-mm-dd hh:mm:ss)
  • audit_period_beginning_max: datetime (yyyy-mm-dd hh:mm:ss)
  • audit_period_ending_min: datetime (yyyy-mm-dd hh:mm:ss)
  • audit_period_ending_max: datetime (yyyy-mm-dd hh:mm:ss)
  • launched_at_min: datetime (yyyy-mm-dd hh:mm:ss)
  • launched_at_max: datetime (yyyy-mm-dd hh:mm:ss)
  • deleted_at_min: datetime (yyyy-mm-dd hh:mm:ss)
  • deleted_at_max: datetime (yyyy-mm-dd hh:mm:ss)
  • received_min: datetime (yyyy-mm-dd hh:mm:ss)
  • received_max: datetime (yyyy-mm-dd hh:mm:ss)
  • instance: uuid
  • limit: int, default: 50, max: 1000
  • offset: int, default: 0

Example request:

GET /db/usage/nova/exists/ HTTP/1.1
Host: example.com
Accept: application/json

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

{
  "exists":
  [
    {
      "status": "verified",
      "os_distro": "org.centos",
      "bandwidth_public_out": 0,
      "received": "2014-01-17 16:16:43.695474",
      "instance_type_id": "2",
      "raw": 14615544,
      "os_architecture": "x64",
      "rax_options": "0",
      "audit_period_ending": "2014-01-17 16:16:43",
      "deleted_at": null,
      "id": 135106,
      "tenant": "5889124",
      "audit_period_beginning": "2014-01-17 00:00:00",
      "fail_reason": null,
      "instance": "978b32ea-374b-48c6-814b-bb6151e2fb5c",
      "instance_flavor_id": "2",
      "launched_at": "2014-01-17 16:16:09",
      "os_version": "6.0",
      "usage": 91932,
      "send_status": 201,
      "message_id": "9d28fa15-d163-40c7-8195-2853ad13179b",
      "delete": null
    },
    {
      "status": "verified",
      "os_distro": "org.centos",
      "bandwidth_public_out": 0,
      "received": "2014-01-17 16:10:42.112505",
      "instance_type_id": "2",
      "raw": 14615459,
      "os_architecture": "x64",
      "rax_options": "0",
      "audit_period_ending": "2014-01-17 16:10:42",
      "deleted_at": null,
      "id": 135105,
      "tenant": "5824940",
      "audit_period_beginning": "2014-01-17 00:00:00",
      "fail_reason": null,
      "instance": "860b5df0-d58b-498d-8838-7156d701732c",
      "instance_flavor_id": "2",
      "launched_at": "2014-01-17 16:10:08",
      "os_version": "5.9",
      "usage": 91937,
      "send_status": 201,
      "message_id": "0a6b1c58-8443-4788-ac08-05cd03e6be53",
      "delete": null
    }
  ]
}

db/usage/glance/exists/

Returns a list of instance exists matching provided query criteria.

Query Parameters

  • audit_period_beginning_min: datetime (yyyy-mm-dd hh:mm:ss)
  • audit_period_beginning_max: datetime (yyyy-mm-dd hh:mm:ss)
  • audit_period_ending_min: datetime (yyyy-mm-dd hh:mm:ss)
  • audit_period_ending_max: datetime (yyyy-mm-dd hh:mm:ss)
  • created_at_min: datetime (yyyy-mm-dd hh:mm:ss)
  • created_at_max: datetime (yyyy-mm-dd hh:mm:ss)
  • deleted_at_min: datetime (yyyy-mm-dd hh:mm:ss)
  • deleted_at_max: datetime (yyyy-mm-dd hh:mm:ss)
  • received_min: datetime (yyyy-mm-dd hh:mm:ss)
  • received_max: datetime (yyyy-mm-dd hh:mm:ss)
  • limit: int, default: 50, max: 1000
  • offset: int, default: 0

Example request:

GET /db/usage/glance/exists/ HTTP/1.1
Host: example.com
Accept: application/json

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

{
  "exists":
  [
    {
      "status": "verified",
      "audit_period_beginning": "2014-01-13 00:00:00",
      "fail_reason": null,
      "uuid": "d39a04bd-6ba0-4d20-8591-937ab43897dc",
      "usage": 2553,
      "created_at": "2013-05-11 15:37:34",
      "size": 11213393920,
      "owner": "389886",
      "message_id": "9c5fd5af-60b4-45ad-b524-c4a9964f31e4",
      "raw": 283303,
      "audit_period_ending": "2014-01-13 23:59:59",
      "received": "2014-01-13 09:20:02.777965",
      "deleted_at": null,
      "send_status": 0,
      "id": 5301,
      "delete": null
    },
    {
      "status": "verified",
      "audit_period_beginning": "2014-01-13 00:00:00",
      "fail_reason": null,
      "uuid": "6713c136-0555-4a93-b726-edb181d4b69e",
      "usage": 1254,
      "created_at": "2013-05-11 15:37:56",
      "size": 11254732800,
      "owner": "389886",
      "message_id": "9c5fd5af-60b4-45ad-b524-c4a9964f31e4",
      "raw": 283303,
      "audit_period_ending": "2014-01-13 23:59:59",
      "received": "2014-01-13 09:20:02.777965",
      "deleted_at": null,
      "send_status": 0,
      "id": 5300,
      "delete": null
    }
  ]
}

db/usage/exists/<exist_id>/

Deprecated, see: dbapi-nova-exist

db/usage/nova/exists/<exist_id>/

Returns a single instance exists matching provided id

Example request:

GET /db/usage/nova/exists/135105/ HTTP/1.1
Host: example.com
Accept: application/json

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

{
  "exist":
  {
    "status": "verified",
    "os_distro": "org.centos",
    "bandwidth_public_out": 0,
    "received": "2014-01-17 16:10:42.112505",
    "instance_type_id": "2",
    "raw": 14615459,
    "os_architecture": "x64",
    "rax_options": "0",
    "audit_period_ending": "2014-01-17 16:10:42",
    "deleted_at": null,
    "id": 135105,
    "tenant": "5824940",
    "audit_period_beginning": "2014-01-17 00:00:00",
    "fail_reason": null,
    "instance": "860b5df0-d58b-498d-8838-7156d701732c",
    "instance_flavor_id": "2",
    "launched_at": "2014-01-17 16:10:08",
    "os_version": "5.9",
    "usage": 91937,
    "send_status": 201,
    "message_id": "0a6b1c58-8443-4788-ac08-05cd03e6be53",
    "delete": null
  }
}

db/usage/glance/exists/<exist_id>/

Returns a single instance exists matching provided id

Example request:

GET /db/usage/glance/exists/5300/ HTTP/1.1
Host: example.com
Accept: application/json

Example response:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json

{
  "exist":
  {
    "status": "verified",
    "audit_period_beginning": "2014-01-13 00:00:00",
    "fail_reason": null,
    "uuid": "6713c136-0555-4a93-b726-edb181d4b69e",
    "usage": 1254,
    "created_at": "2013-05-11 15:37:56",
    "size": 11254732800,
    "owner": "389886",
    "message_id": "9c5fd5af-60b4-45ad-b524-c4a9964f31e4",
    "raw": 283303,
    "audit_period_ending": "2014-01-13 23:59:59",
    "received": "2014-01-13 09:20:02.777965",
    "deleted_at": null,
    "send_status": 0,
    "id": 5300,
    "delete": null
  }
}

/db/repair

Changes the status of all the exists of message-ids sent with the request from 'pending' to 'sent_unverified' so that the verifier does not end up sending .verified for all those exists(since the .exists have already been modified as .verified and sent to AH by Yagi). It sends back the message-ids of exists which could not be updated in the json response.

Example request:

POST /db/repair/  HTTP/1.1
Host: example.com
Accept: application/json

**Example response**:
HTTP/1.1 200 OK
Vary: Accept
Content-Type: text/json

{
u'exists_not_pending': [u'494ebfce-0219-4b62-b810-79039a279620'],
u'absent_exists': [u'7609f3b2-3694-4b6f-869e-2f13ae504cb2',
                   u'0c64032e-4a60-44c0-a99d-5a4f2e46afb0']
}
query message_ids

list of message_ids of exists messages

query service

nova or glance. default="nova"