Redmine core has a separate permission for viewing logged time. When this is not active for a specific role (e.g. clients), they can still access it as a card item in the agile board.
The plugin should adhere to the existing permissions and not give the user access to spent time when his role doesn't permit so.

Similar, we would like to be able to hide estimated hours (core field) as wel for certain roles.
We are using redmine_base_deface in combination with redmine_customize_core_fields to hide it on the issues for now, but also here, this can still be accessed via the agile board.