When a dataset TTL is updated, then the updated TTL value is not propagated to the underlying HBase table's table descriptor. This prevents the Tephra co-processor from getting the updated value of the TTL during major compactions, and thus the new TTL will not be applied.
This regression was introduced during a re-factoring in release 4.3.1 - https://github.com/cdapio/cdap/blob/v5.1.3/cdap-data-fabric/src/main/java/co/cask/cdap/data2/dataset2/lib/table/hbase/HBaseTableAdmin.java#L177. In addition to TTL, some other properties that influence readless increments, bloom filter and max values are also not propagated.
Note that, during creation of a table the properties are propagated correctly.