Skip to content

Commit

Permalink
Summary filter (#960)
Browse files Browse the repository at this point in the history 
* feat: implement exponents in QA

* feat: add filter for submission values

* chore: remove duplicated property

* feat: implement value search and hotlinking

this branch replaces the work on
#948

* feat: use expression builder for value search

* feat: use tom-select for first field dropdown

* feat: use custom dropdown on value

* chore: remove comment

* chore: switch to selectize from tom-select

* chore: remove tom-select artifacts

* chore: don't use label in dropdown

* fix: destroy selectize control when source changes to value

* chore: add destroyed hook

* feat: after changing source, warn for incomplete term

* feat: clean up selectize when term is removed.

* chore: various UI changes

- remove icon on button
- change button text color

* fix: don't destroy all the dropdowns

* fix: destroy all selectize instances on expression deletion

* feat: start using new template

* feat: working with new template

* chore: change delete button visibility rule

this commit changes the delete term button display rule.
now, it displays on each term unless there's only one available

* chore: remove unused code

* fix: correct issue with locations import (#979)

* chore(deps): bump gettext.js from 0.8.0 to 2.0.3 in /apollo/static (#978)

Bumps gettext.js from 0.8.0 to 2.0.3.

---
updated-dependencies:
- dependency-name: gettext.js
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: fix PWA issues (#980)

this commit fixes issues reported from users.
the issues are due to upgrading marshmallow

* chore(deps-dev): bump webpack from 5.76.0 to 5.94.0 in /apollo/static (#982)

Bumps [webpack](https://github.com/webpack/webpack) from 5.76.0 to 5.94.0.
- [Release notes](https://github.com/webpack/webpack/releases)
- [Commits](webpack/webpack@v5.76.0...v5.94.0)

---
updated-dependencies:
- dependency-name: webpack
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: replace all usage of _joined_entities() (#981)

* fix: replace all usage of _joined_entities()

SQLAlchemy 1.4 removed an internal query method (_joined_entities())
that was used to check if a query was already joined to a model.
This pull request replaces the usages of that method.

* fix: add missing file

* feat: add unit tests and comment

* fixed a few nits

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tim Akinbo <[email protected]>
  • Loading branch information
@dodumosu @dependabot @takinbo
3 people authored Sep 9, 2024
1 parent 4d1ef30 commit f561f33
Show file tree
Hide file tree
Showing 9 changed files with 290 additions and 18 deletions.
4 changes: 2 additions & 2 deletions apollo/formsframework/models.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -151,11 +151,11 @@ def unsorted_tags(self):
return list(self._field_cache.keys())

@property
def unsorted_tags(self):
def response_fields(self):
if not hasattr(self, '_field_cache'):
self._populate_field_cache()

return list(self._field_cache.keys())
return list(self._field_cache.values())

@property
def qa_tags(self):
Expand Down
38 changes: 37 additions & 1 deletion apollo/frontend/templates/frontend/macros/submission_list_filter.html
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{% macro render_filter_form(form, filter_form, location) %}
<div class="card border-light bg-light mb-3">
<div class="card border-light bg-light mb-3" id="filter-app">
<div class="card-header">
<form class="mb-n2 ml-n2 mr-n2">
{% if form.show_map %}<input type="hidden" id="v" name="v" value="{{ request.args.v }}">{% endif %}
Expand Down Expand Up @@ -85,6 +85,42 @@
</div>
</div>
{%- endif %}
<div class="col-12 col-lg-12" id="valuesWrapper" data-form-schema='{{ form.response_fields|tojson }}'>
<div class="input-group d-flex mb-2" v-for="(term, index) in terms">
<select class="form-control custom-select" v-model="term.field">
<option disabled :value="null">{{ _('Select Field') }}</option>
<option v-for="(field, idx) in fieldOptions" :key="idx" :value="field.value">[[ field.label ]]</option>
</select>
<select class="form-control custom-select" v-model="term.operator">
<option disabled :value="null">{{ _('Select Operator') }}</option>
<option value="=">=</option>
<option value="<>">&lt;&gt;</option>
<option value="<" v-if="term.field && term.field.type === 'integer'">&lt;</option>
<option value="<=" v-if="term.field && term.field.type === 'integer'">&lt;=</option>
<option value=">" v-if="term.field && term.field.type === 'integer'">&gt;</option>
<option value=">=" v-if="term.field && term.field.type === 'integer'">&gt;=</option>
</select>
<select class="form-control custom-select" v-model="term.source" v-if="term.field !== null && term.field.type === 'integer'">
<option disabled value="">{{ _('Select Source') }}</option>
<option value="field">Field</option>
<option value="value">Value</option>
</select>
<input type="number" class="form-control" v-model="term.value" v-if="term.field !== null && term.field.type === 'integer' && term.source === 'value'">
<select class="form-control custom-select" v-model="term.value" v-if="term.field !== null && term.field.type === 'integer' && term.source === 'field'">
<option disabled :value="null">{{ _('Select Field') }}</option>
<option :value="fi.value" v-for="(fi, indx) in getValueFields(index)" :key="indx">[[ fi.label ]]</option>
</select>
<select class="form-control custom-select" v-model="term.value" v-if="term.field && term.field.type === 'select'">
<option disabled :value="null">{{ _('Select Value') }}</option>
<option v-for="[key, value] in getSortedOptions(term.field.options)" :value="value" :key="value">[[ value ]]</option>
</select>
<div class="input-group-append">
<button class="btn btn-success" @click.prevent="addNewTerm" aria-label="{{ _('Add Search Term') }}" v-if="index === (terms.length - 1)"><i class="fa fa-plus-circle"></i></button>
<button class="btn btn-danger" @click.prevent="removeItem(index)" aria-label="{{ _('Delete Search Term') }}" v-if="terms.length > 1"><i class="fa fa-minus-circle"></i></button>
</div>
</div>
<input type="hidden" name="{{ filter_form.values.name }}" id="{{ filter_form.values.id }}" :value="serializedExpressions">
</div>
<div class="col-12 col-md-6 col-lg-2 mb-2">
<div class="d-flex flex-row">
<button class="btn btn-primary mr-2 flex-fill" type="submit">{{ _('Filter') }}</button>
Expand Down
195 changes: 194 additions & 1 deletion apollo/frontend/templates/frontend/submission_list.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

{% block stylesheets %}
<link rel="stylesheet" href="{{ url_for('static', filename='css/bootstrap-switch-button.min.css') }}">
<link rel="stylesheet" href="{{ url_for('static', filename='css/selectize.bootstrap4.css') }}">
<link rel="stylesheet" href="{{ asset_url_for('datetimepicker.css') }}">
<link rel="stylesheet" href="{{ asset_url_for('leaflet.css') }}">
<style>
Expand All @@ -14,13 +15,205 @@
{% endblock %}
{% block scripts -%}
<script src="{{ url_for('static', filename='js/bootstrap-switch-button.min.js') }}"></script>
<script src="{{ url_for('static', filename='js/browser-file-storage.min.js') }}"></script>
<script src="{{ url_for('static', filename='js/jszip.min.js') }}"></script>
<script src="{{ url_for('static', filename='js/ohm.min.js') }}"></script>
<script src="{{ url_for('static', filename='js/selectize.js') }}"></script>
<script src="{{ url_for('static', filename='js/vue.min.js') }}"></script>
<script src="{{ url_for('static', filename='js/browser-file-storage.min.js') }}"></script>
<script type="text/javascript" src="{{ asset_url_for('moment.js') }}" charset="utf-8"></script>
<script type="text/javascript" src="{{ asset_url_for('datetimepicker.js') }}" charset="utf-8"></script>
<script type="text/javascript" src="{{ asset_url_for('leaflet.js') }}" charset="utf-8"></script>
<script type="text/javascript">
$(function () {
// initialize the parser
const formSchema = JSON.parse(document.querySelector('#valuesWrapper').dataset.formSchema);

const choiceFields = formSchema.filter(f => f.type === 'select').reduce((accumulator, f) => {
accumulator[f.tag] = f;
return accumulator
}, {});

const integerFields = formSchema.filter(f => f.type === 'integer').reduce((accumulator, f) => {
accumulator[f.tag] = f;
return accumulator
}, {});

const usableFields = {...choiceFields, ...integerFields};

let choiceFieldsGrammar, integerFieldsGrammar, choiceActions, intActions, choiceSemantics, intSemantics;
if (Object.keys(choiceFields).length > 0) {
choiceFieldsGrammar = ohm.grammar(`
QueryItem {
expression = field spaces operator spaces number
number = digit+
operator = "=" | "!=" | "<>"
field = ${Object.keys(choiceFields).map(f => `"${f}"`).join(" | ")}
}
`);
choiceActions = {
expression(field, _1, operator, _2, number) {
return {
field: choiceFields[field.eval()],
operator: operator.eval(),
value: number.eval(),
source: "value"
};
},
operator(o) {
return o.sourceString;
},
field(f) {
return f.sourceString;
},
number(digits) {
return parseInt(digits.sourceString);
}
};
choiceSemantics = choiceFieldsGrammar.createSemantics();
choiceSemantics.addOperation('eval', choiceActions);
}

if (Object.keys(integerFields).length > 0) {
integerFieldsGrammar = ohm.grammar(`
QueryItem {
expression = field spaces operator spaces entity
entity = field | number
number = digit+
operator = "=" | "!=" | "<>" | ">=" | ">" | "<=" | "<"
field = ${Object.keys(integerFields).map(f => `"${f}"`).join(" | ")}
}
`);

intActions = {
expression(field, _1, operator, _2, entity) {
data = {
field: integerFields[field.eval()],
operator: operator.eval(),
value: entity.eval()
};

if (typeof data["value"] === "number")
data["source"] = "value";
else
data["source"] = "field";

return data;
},
entity(e) {
return e.eval();
},
operator(o) {
return o.sourceString;
},
field(f) {
return f.sourceString;
},
number(digits) {
return parseInt(digits.sourceString);
}
};

intSemantics = integerFieldsGrammar.createSemantics();
intSemantics.addOperation('eval', intActions);
}

const queryParams = new URLSearchParams(window.location.search);
const valueParams = queryParams.get('values');
const expressions = (valueParams || '').split(',');

let choiceData, integerData;

if (choiceFieldsGrammar) {
choiceData = expressions.filter(
e => choiceFieldsGrammar.match(e).succeeded()
).map(
e => choiceSemantics(choiceFieldsGrammar.match(e)).eval()
);
} else {
choiceData = [];
}

if (integerFieldsGrammar) {
integerData = expressions.filter(
e => integerFieldsGrammar.match(e).succeeded()
).map(
e => intSemantics(integerFieldsGrammar.match(e)).eval()
);
} else {
integerData = [];
}

const initialData = [...choiceData, ...integerData];

const app = new Vue({
computed: {
expressions() {
const thisRef = this;
const completeTerms = this.terms.filter(term => thisRef.isComplete(term));
return completeTerms.map(term => `${term.field.tag}${term.operator}${term.value}`);
},
fieldMap() {
return usableFields;
},
fieldOptions() {
return Object.keys(usableFields).sort().map(i => {
return {label: i, value: usableFields[i]};
});
},
serializedExpressions() {
const thisRef = this;
const completeTerms = this.terms.filter(term => thisRef.isComplete(term));
return completeTerms.map(term => `${term.field.tag}${term.operator}${term.value}`).join(',');
},
},
data: {
schema: formSchema,
terms: [...initialData]
},
delimiters: ['[[', ']]'],
el: '#filter-app',
methods: {
addNewTerm() {
this.terms.push({
field: null,
operator: null,
value: null,
source: null
});
},
getSortedOptions(options) {
if (options === undefined)
return [];

const pairs = Object.entries(options).toSorted((a, b) => a[1] - b[1]);
return pairs;
},
getValueFields(index) {
const thisRef = this;
if (!thisRef.terms[index].field || thisRef.terms[index].field.type !== 'integer') return [];

const tags = thisRef.schema.filter(f => f.tag !== thisRef.terms[index].field.tag && f.type === 'integer').map(f => f.tag);
return tags.filter(f => f !== thisRef.terms[index].field.tag).map(f => {
return {label: f, value: f};
});
},
isComplete(term) {
return !this.isEmpty(term.field) && !this.isEmpty(term.operator) && !this.isEmpty(term.value);
},
isEmpty(val) {
return (val === null) || (val === undefined) || (val === "");
},
removeItem(index) {
this.terms.splice(index, 1);
}
},
mounted() {
if (this.terms.length === 0) {
this.addNewTerm();
}
}
});

// initialize the toast
$('#notification-toast').toast();

Expand Down
2 changes: 1 addition & 1 deletion apollo/process_analysis/templates/process_analysis/checklist_summary.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ <h4 class="font-weight-light mt-4{{ ' rtl' if g.locale.text_direction == 'rtl' e
{%- if field['type'] == 'integer' %}
<th>{{ value }}</th>
{%- else %}
<th>{{ _('%(label)s', label=label) }} ({{ value }})</th>
<th><a href="{{ url_for('submissions.submission_list', form_id=form.id, values=filter_option_value(field['tag'], '=', value)) }}">{{ _('%(label)s', label=label) }} ({{ value }})</a></th>
{%- endif %}
{% endfor %}
{% elif stats['type'] == 'bucket' %}
Expand Down
4 changes: 4 additions & 0 deletions apollo/process_analysis/templates/process_analysis/checklist_summary_breakdown.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,11 @@
{% elif tag_stats.type == 'histogram' %}
{% for label, option in tag_stats.meta %}
{%- if tag_stats.labels %}
{% if field.type == 'integer' %}
<th colspan="2">{{ _('%(label)s', label=label) }} ({{ option }})</th>
{% else %}
<th colspan="2"><a href="{{ url_for('submissions.submission_list', form_id=form.id, values=filter_option_value(field['tag'], '=', option)) }}">{{ _('%(label)s', label=label) }} ({{ option }})</a></th>
{% endif %}
{%- else %}
<th colspan="2">{{ option }}</th>
{% endif %}
Expand Down
7 changes: 7 additions & 0 deletions apollo/process_analysis/views_process.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
from collections import OrderedDict
from functools import partial
from operator import itemgetter
from string import Template

from flask import Blueprint, g, render_template, request, url_for
from flask_babelex import gettext as _
Expand All @@ -24,6 +25,11 @@
from apollo.submissions.utils import make_submission_dataframe


def filter_option_value(tag: str, op: str, value: str) -> str:
s = Template('$tag$op$value')
return s.substitute(tag=tag, op=op, value=value)


def get_analysis_menu():
event = g.event
subquery = or_(
Expand Down Expand Up @@ -175,6 +181,7 @@ def _process_analysis(event, form_id, location_id=None, tag=None):
context['field_groups'] = OrderedDict()
context['navigation_data'] = analysis_navigation_data(
form, location, display_tag)
context['filter_option_value'] = filter_option_value

# processing for incident forms
if form.form_type == 'INCIDENT':
Expand Down
1 change: 1 addition & 0 deletions apollo/static/js/ohm.min.js
View file

Large diffs are not rendered by default.

23 changes: 22 additions & 1 deletion apollo/submissions/filters.py
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# -*- coding: utf-8 -*-
from cgi import escape
from itertools import chain
from operator import itemgetter

from cgi import escape
from dateutil.parser import parse
from dateutil.tz import gettz, UTC
from flask_babelex import gettext as _
Expand Down Expand Up @@ -640,6 +640,26 @@ def queryset_(self, queryset, value):
return queryset


class SubmissionValuesFilter(CharFilter):
def __init__(self, name=None, widget=None, **kwargs):
self.form = kwargs.pop('questionnaire', None)
super().__init__(name, widget, **kwargs)

def queryset_(self, queryset, value, **kwargs):
if value:
if self.form is None:
return queryset.filter(false())

try:
parsed = value.split(',')
terms = [generate_qa_query(item, self.form)[0] for item in parsed]
except Exception:
return queryset.filter(false())

return queryset.filter(*terms)
return super().queryset_(queryset, value, **kwargs)


def make_submission_location_filter(location_set_id):
class AJAXLocationFilter(ChoiceFilter):
field_class = LocationQuerySelectField
Expand Down Expand Up @@ -749,6 +769,7 @@ def make_submission_list_filter(event, form, filter_on_locations=False):
attributes['fsn'] = FormSerialNumberFilter()
attributes['participant_role'] = make_participant_role_filter(
event.participant_set_id)()
attributes['values'] = SubmissionValuesFilter(questionnaire=form)

return type(
'SubmissionFilterSet',
Expand Down
Loading

0 comments on commit f561f33

Please sign in to comment.